From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Fri May 21 2004 - 02:28:15 MDT
Michael Roy Ames wrote:
> Keith,
>
> Samantha wrote:
>
>>>What does logic or your feelings have to do with it?
>>>We are fast developing the ability to reprogram
>>>ourselves.
>
> Then Keith replied:
>
>>Without the utmost caution, that way lies oblivion.
>
> Gee, Keith, are you playing the harbinger of doom? I am eagerly looking
> forward to reprogramming some of the counterproductive goals/desires in my
> own mind... for a trivial example: my desire to consume large quantities of
> bacon, cheese, chocolate and wine. Why would 'dialing down' my hunger for
> fat, sugar and drugs lead me to doom?
I think I have to side with Keith. I fear that human self-modification
is far more dangerous than I would once have liked to imagine. Better
to devise nutritious bacon, cheese, chocolate, and wine, than dare to
mess with hunger - let alone anything more complex. You would
practically need to be a Friendly AI programmer just to realize how
afraid you needed to be, and freeze solid until there was an AI midwife
at hand to help you *very slowly* start to make modifications that
didn't have huge unintended consequences, or take you away from the rest
of humanity, or destroy complexity you would have preferred to keep.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:36 MST