Re: SIAI's flawed friendliness analysis

From: Eliezer S. Yudkowsky (
Date: Fri May 16 2003 - 21:17:43 MDT

Bill Hibbard wrote:
> The ambiguous definitions in the SIAI analysis will be
> exploited by powerful people and institutions to create
> AIs that protect and enhance their own interests.

I am not going to say that the problem of third-party enforcement of
Friendliness, given a programmer actively trying to twist the regulations,
is "un-solvable". I have learned better than to use words like that. The
word "impossible" means "I do not know how to do this right now; ask me
again tomorrow."

Bearing that in mind: the problem you describe is impossible.

I will be ecstatic if the problem of Friendliness is solvable by someone
who genuinely wants to solve it. The problem given a hostile programmer
is... how can I put this... INSANE. If you want technological advancement
on the problem of trust, build a better lie detector using advances in
neuroimaging, signal processing, and pattern recognition, and run the
programmers through it. I hereby volunteer

Eliezer S. Yudkowsky                
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:42 MDT