Re: Flawed Risk Analysis (was Re: SIAI's flawed friendliness analysis)

From: Bill Hibbard (
Date: Fri May 23 2003 - 10:18:37 MDT

On Wed, 21 May 2003 wrote:

> Uh, NO! The flaw in your argument is you say that if human designers
> can create a very primitive, infantile AI, then human regulators can
> inspect that AI efficently three years down the road when it's ten
> times smarter than us! Does anyone besides me see that gaping logical
> flaw?

I never said what follows your "you say ...".

We don't have to follow an AI's detailed thoughts. The
inspection is of the design, not the changing contents
of its mind. If it's initial reinforcement values are
for human happiness, and its simulation and reinforcement
learning algorithms are accurate, then we can trust the
way it will develop. In an earlier email I made the
analogy to game playing programs. If their game
simulation and learning algorithms are accurate and
efficient, and their reinforcement learning values are
for winning the game, then although the details of
their play are not predictable, the fact that they will
play to win is predictable.

Bill Hibbard, SSEC, 1225 W. Dayton St., Madison, WI 53706 608-263-4427 fax: 608-263-6738

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:42 MDT