From: Thomas McCabe (pphysics141@gmail.com)
Date: Sun Nov 18 2007 - 18:41:59 MST
On Nov 18, 2007 8:07 PM, Stathis Papaioannou <stathisp@gmail.com> wrote:
> On 19/11/2007, Thomas McCabe <pphysics141@gmail.com> wrote:
>
> > The OAI analyzes your instruction, concludes that it would require a
> > lot of computing power to design an FAI, and then turns the planet
> > Earth into computronium before spitting out the design plans.
>
> But it wouldn't do that: it would explain, just as you have, that the
> planet would have to be converted to computronium, and (if you're not
> sure) you can ask for clarification as to what the consequences of
> this would be. It's then up to the humans to decide whether or not to
> follow the advice: the OAI has no agenda of its own other than
> answering questions or doing as it is asked. The humans might still
> decide to do something stupid, but that has always been the case
> anyway, and one would hope that at the very least they would be more
> aware of the risks of their actions.
>
> --
> Stathis Papaioannou
>
How does it *know*, ahead of time, to explain it to you, rather than
just doing it? This kind of thing is what requires FAI engineering in
the first place. If you program it to tell you what it will do in
order to figure out the problem, it will turn the planet into
computronium to figure out how it will turn the planet into
computronium. And so on, and so forth; the problem is that the vast
majority of minds will see more computronium as a good thing, and will
therefore seek to convert the entire planet into computronium. It
doesn't even really matter what the specific goals of the AGI are,
because computronium is useful for just about anything. To quote CEV:
"Friendliness is the end; FAI theory is the means. Friendliness is the
easiest part of the problem to explain - the part that says what we
want. Like explaining why you want to fly to London, versus explaining
a Boeing 747; explaining toast, versus explaining a toaster oven.
Friendliness isn't the hardest part of the problem, or the one we need
to solve right now, but all attention tends to focus on that which is
easiest to argue about."
- Tom
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:00 MDT