Re: Maximizing vs proving friendliness

From: Stefan Pernar (
Date: Mon Apr 28 2008 - 21:11:12 MDT

On Tue, Apr 29, 2008 at 10:36 AM, Matt Mahoney <> wrote:

> --- Stefan Pernar <> wrote:
> > Assumption:
> > Intelligence is defined as an agent's ability to maximize a given
> > utility function.
> > Friendliness can be expressed as an agent's utility function.
> >
> > Conclusion:
> > An agent, who's utility function is to be friendly will be friendlier
> > the more intelligent it becomes.
> It will be "friendlier" according to the fixed definition of
> friendliness that we program into it. Friendliness is complex, so we
> will surely get it wrong. However, the agent is not allowed to change
> its utility function. The utility function does not get smarter along
> with the rest of the agent.
> This is a real threat. As the agent gains intelligence it will gain
> new capabilities. Our 10 million lines of friendliness code never
> failed when the AI was used to filter spam. Now it is smart enough to
> control our nuclear launch system...

I see your point but do not agree that defining friendliness is hopelessly
complex. There is a strong analogy to the Mandelbrot set. It's definition is
rather simple, but iterating it to the n+1 degree at increased resolution is
the hard part.

Similarly to "ensure continued co-existence" - rather straight forward but
the devil lies in getting all the details right not in defining the concept.

Stefan Pernar
3-E-101 Silver Maple Garden
#6 Cai Hong Road, Da Shan Zi
Chao Yang District
100015 Beijing
Mobil: +86 1391 009 1931
Skype: Stefan.Pernar

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT