Re: FAI means no programmer-sensitive AI morality

From: Eliezer S. Yudkowsky (
Date: Sat Jun 29 2002 - 09:38:13 MDT

Samantha Atkins wrote:
> Ben Goertzel wrote:
>>> But it should be equally *true* for every individual, whether or not
>>> the individual realizes it in advance, that they have nothing to fear
>>> from the AI being influenced by the programmers. An AI programmer
>>> should be able to say to anyone, whether atheist, Protestant,
>>> Catholic, Buddhist, Muslim, Jew, et cetera: "If you are right and I
>>> am wrong then the AI will agree with you, not me."
> Of course some breeds of religious people would simply claim that unless
> the AI has an immortal soul (or Buddha nature) and it is capable of
> communion with the Holy Ghost or some such that it cannot know about
> these religious matters at all. As you say below, a non-empiricist
> element.

I don't see why a Friendly AI would have trouble handling that case in the
event it turned out to be true. I say "Show kindness toward all sentient
creatures" and "Hunt down the reasons I made that statement". The AI grows
up, scans my physical brainstate, and finds that I have an immortal soul
which contributed to the generation of that statement. At which point the
AI could pray for an immortal soul from God, merge with a human so that the
combined entity would have an immortal soul, say "AI? Forget it! Wrong
future!" and terminate after constructing a few human-enhancement kiosks, et

We are perhaps extraordinarily unlikely to find ourselves in that particular
strange situation, but might very well end up in one even stranger.

Eliezer S. Yudkowsky                
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:39 MDT