From: Bill Hibbard (test@demedici.ssec.wisc.edu)
Date: Thu Jun 08 2006 - 02:50:34 MDT
On Wed, 7 Jun 2006, Robin Lee Powell wrote:
> On Wed, Jun 07, 2006 at 12:24:55PM -0500, Bill Hibbard wrote:
> > If you think RL can succeed at intelligence but must fail at
> > friendliness, but just want to demonstrate it for a specific
> > example, then use a scenario in which:
> >
> > 1. The SI recognizes humans and their emotions as accurately as
> > any human, and continually relearns that recognition as humans
> > evolve (for example, to become SIs themselves).
> >
> > 2. The SI values people after death at the maximally unhappy
> > value, in order to avoid motivating the SI to kill unhappy
> > people.
> >
> > 3. The SI combines the happiness of many people in a way (such
> > as by averaging) that does not motivate a simple numerical
> > increase (or decrease) in the number of people.
> >
> > 4. The SI weights unhappiness stronger than happiness, so that
> > it focuses it efforts on helping unhappy people.
> >
> > 5. The SI develops models of all humans and what produces
> > long-term happiness in each of them.
> >
> > 6. The SI develops models of the interactions among humans and
> > how these interactions affect the happiness of each.
>
> Have you read The Metamorphosis Of Prime Intellect?
>
> The scenario above immediately and obviously falls to the "I've
> figured out where human's pleasure centers are; I'll just leave them
> on" failure.
I address this issue in my 2005 on-line paper:
The Ethics and Politics of Super-Intelligent Machines
http://www.ssec.wisc.edu/~billh/g/SI_ethics_politics.doc
There exists a form of happiness that is not drug-induced
ecstasy.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT