From: Stefan Pernar (stefan.pernar@gmail.com)
Date: Mon Apr 28 2008 - 21:11:12 MDT
On Tue, Apr 29, 2008 at 10:36 AM, Matt Mahoney <matmahoney@yahoo.com> wrote:
> --- Stefan Pernar <stefan.pernar@gmail.com> wrote:
> > Assumption:
> > Intelligence is defined as an agent's ability to maximize a given
> > utility function.
> > Friendliness can be expressed as an agent's utility function.
> >
> > Conclusion:
> > An agent, who's utility function is to be friendly will be friendlier
> > the more intelligent it becomes.
>
> It will be "friendlier" according to the fixed definition of
> friendliness that we program into it. Friendliness is complex, so we
> will surely get it wrong. However, the agent is not allowed to change
> its utility function. The utility function does not get smarter along
> with the rest of the agent.
>
> This is a real threat. As the agent gains intelligence it will gain
> new capabilities. Our 10 million lines of friendliness code never
> failed when the AI was used to filter spam. Now it is smart enough to
> control our nuclear launch system...
>
I see your point but do not agree that defining friendliness is hopelessly
complex. There is a strong analogy to the Mandelbrot set. It's definition is
rather simple, but iterating it to the n+1 degree at increased resolution is
the hard part.
Similarly to "ensure continued co-existence" - rather straight forward but
the devil lies in getting all the details right not in defining the concept.
-- Stefan Pernar 3-E-101 Silver Maple Garden #6 Cai Hong Road, Da Shan Zi Chao Yang District 100015 Beijing P.R. CHINA Mobil: +86 1391 009 1931 Skype: Stefan.Pernar
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT