From: Robin Lee Powell (rlpowell@digitalkingdom.org)
Date: Thu Jun 08 2006 - 11:32:27 MDT
On Thu, Jun 08, 2006 at 03:50:34AM -0500, Bill Hibbard wrote:
> On Wed, 7 Jun 2006, Robin Lee Powell wrote:
>
> > On Wed, Jun 07, 2006 at 12:24:55PM -0500, Bill Hibbard wrote:
> > > If you think RL can succeed at intelligence but must fail at
> > > friendliness, but just want to demonstrate it for a specific
> > > example, then use a scenario in which:
> > >
> > > 1. The SI recognizes humans and their emotions as accurately
> > > as any human, and continually relearns that recognition as
> > > humans evolve (for example, to become SIs themselves).
> > >
> > > 2. The SI values people after death at the maximally unhappy
> > > value, in order to avoid motivating the SI to kill unhappy
> > > people.
> > >
> > > 3. The SI combines the happiness of many people in a way
> > > (such as by averaging) that does not motivate a simple
> > > numerical increase (or decrease) in the number of people.
> > >
> > > 4. The SI weights unhappiness stronger than happiness, so
> > > that it focuses it efforts on helping unhappy people.
> > >
> > > 5. The SI develops models of all humans and what produces
> > > long-term happiness in each of them.
> > >
> > > 6. The SI develops models of the interactions among humans
> > > and how these interactions affect the happiness of each.
> >
> > Have you read The Metamorphosis Of Prime Intellect?
> >
> > The scenario above immediately and obviously falls to the "I've
> > figured out where human's pleasure centers are; I'll just leave
> > them on" failure.
>
> I address this issue in my 2005 on-line paper:
>
> The Ethics and Politics of Super-Intelligent Machines
> http://www.ssec.wisc.edu/~billh/g/SI_ethics_politics.doc
>
> There exists a form of happiness that is not drug-induced ecstasy.
I read all of the paragraphs with the word "happiness" in them. I
see nothing that addresses this issue even in the slightest.
-Robin
-- http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/ Reason #237 To Learn Lojban: "Homonyms: Their Grate!" Proud Supporter of the Singularity Institute - http://intelligence.org/
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT