From: Mark Waser (mwaser@cox.net)
Date: Sun May 30 2004 - 07:53:03 MDT
> I am at a loss to understand what is gained, except a way of
> anthropomorphizing the issue and avoiding confrontation with that darned
> hard part of the problem.
OK. Try this scenario . . . . An FAI believes that it has a good goal
enhancement. It tests it. It looks OK. It implements it. Circumstances
subsequently change in a REALLY unexpected manner. Oops! Under these very
odd new circumstances, the "enhancement" turns out to be catastrophic for
the human race . . . .
But wait! There are three other FAIs that are in close communication with
this FAI. Friendliness dictates that they should work in close co-operation
in order to prevent errors of this type. The other FAIs have not made this
particular "enhancement" and correctly evaluate the situation for what it
is. They outvote the FAI with the "enhancement" and the "enhanced" FAI (who
is still mostly Friendly and merely mistaken) rolls the "enhancement" back
(or modifies it) so that the human race lives for the next moment or so . .
. .
Good engineering often dictates redundancy. Common sense (which ain't so
common - yes, I know) strongly promotes checks and balances. Human history
shows that when diversity of opinion is allowed to flourish that good things
happen and that when diversity is suppressed that BAD things happen. You
seem to be flying in the face of a lot of good consensus about safety
measures without a reason except "I am at a loss to understand what is
gained . . . ."
> elaborate network of interacting
> optimization processes, threatening each other into not doing certain
> things
Why would one Friendly AI ever threaten another? I would think that they
would work closely and collaboratively to ensure that the best decision is
reached and implemented (or, at least, to NOT implement decisions that seem
unsafe).
Why don't we assume that I'm a Friendly human telling you that I'm pretty
sure that a single point of failure is A REALLY BAD IDEA(tm). I would hope
that you would take this seriously enough that you wouldn't ignore this
advice and implement your plan solely on the basis of "I am at a loss to
understand what is gained . . . ."
By the way, I do understand that you don't acknowledge the distinction
between multiple AIs with close communication and one AI with partitioning
but I would submit that one AI with sufficient partitioning SHOULD BE
considered separate AIs for all intents and purposes. Or, if the
partitioning is not sufficient for them, to be considered separate AIs, then
you need more partitioning in your single AI to create multiple AIs to
prevent the problem above.
> Or perhaps it would help to have three AI projects competing to achieve
the
> threshold of runaway recursive self-improvement, so that whichever one
> takes the fewest safety precautions finishes first.
Nice sarcasm. Not at all conducive to an enlightened debate, but elegantly
done . . . .
Mark
----- Original Message -----
From: "Eliezer Yudkowsky" <sentience@pobox.com>
To: <sl4@sl4.org>
Sent: Sunday, May 30, 2004 3:39 AM
Subject: Re: One or more AIs??
> Which is better for the human species?
>
> One enormously powerful, recursively self-improving runaway optimization
> process trying to tile the solar system with paperclips?
>
> Or three enormously powerful, recursively self-improving runaway
> optimization processes, respectively trying to tile the solar system with
> paperclips, tiny pictures of smiling humans, and reward-button circuitry?
> Maybe they'd fight to the finish, but more likely they'd compromise with
> each other in a way that simultaneously maximized paperclips, tiny
pictures
> of smiling humans, and reward-button circuitry.
>
> To get *any benefit* out of an optimization process, you have to solve the
> technical problems of Friendly AI to locate the tiny "beneficial" subspace
> of the space of optimization processes. It doesn't matter whether you do
> it using one optimization process, or some grandly Rube Goldberg scheme to
> make it unnecessarily conditional on an elaborate network of interacting
> optimization processes, threatening each other into not doing certain
> things - except, of course, that the latter is unnecessarily dangerous and
> less likely to work. It's just as hard whether you call your working
> materials by the irrelevant label, "many" or "one".
>
> Or perhaps it would help to have three AI projects competing to achieve
the
> threshold of runaway recursive self-improvement, so that whichever one
> takes the fewest safety precautions finishes first.
>
> I am at a loss to understand what is gained, except a way of
> anthropomorphizing the issue and avoiding confrontation with that darned
> hard part of the problem.
>
> --
> Eliezer S. Yudkowsky http://intelligence.org/
> Research Fellow, Singularity Institute for Artificial Intelligence
>
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT