Re: Volitional Morality and Action Judgement

From: Eliezer Yudkowsky (
Date: Fri May 21 2004 - 02:28:15 MDT

Michael Roy Ames wrote:
> Keith,
> Samantha wrote:
>>>What does logic or your feelings have to do with it?
>>>We are fast developing the ability to reprogram
> Then Keith replied:
>>Without the utmost caution, that way lies oblivion.
> Gee, Keith, are you playing the harbinger of doom? I am eagerly looking
> forward to reprogramming some of the counterproductive goals/desires in my
> own mind... for a trivial example: my desire to consume large quantities of
> bacon, cheese, chocolate and wine. Why would 'dialing down' my hunger for
> fat, sugar and drugs lead me to doom?

I think I have to side with Keith. I fear that human self-modification
is far more dangerous than I would once have liked to imagine. Better
to devise nutritious bacon, cheese, chocolate, and wine, than dare to
mess with hunger - let alone anything more complex. You would
practically need to be a Friendly AI programmer just to realize how
afraid you needed to be, and freeze solid until there was an AI midwife
at hand to help you *very slowly* start to make modifications that
didn't have huge unintended consequences, or take you away from the rest
of humanity, or destroy complexity you would have preferred to keep.

Eliezer S. Yudkowsky                
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:46 MDT