From: Stuart Armstrong (dragondreaming@googlemail.com)
Date: Fri Jun 27 2008 - 08:19:43 MDT
> Remember, an AI is expected to actually do something, in this case
> produce a Singularity. So the AI must like to do stuff. When you try to
> do stuff there are usually obstacles in your path. The more intelligent
> you are the more you can look at these obstacles from different
> perspectives to find the best way around them.
>
> When Mr. Jupiter Brain encounters a obstruction it won't matter if it
> was placed there by Nature or by Human Beings in a pathetic attempt to
> remain boss; it will contemplate the problem from directions neither you
> or I can imagine and deal with it accordingly.
This is quite interesting. You're viewing the AI's goal of "doing
stuff" are being a fundamental motivating aspect of itself, while the
goal "serve humans" as being a narrow goal, a complex wish that the AI
will manoeuvre round. And to some extent, you're right; if the AI's
mental architecture is set up in that way (we could imagine it has a
"Lawyer module" that looks at the "serve humans" goal and reduces it
until it's nearly meaningless). It's a good warning not to design AI's
that way!
But why do you assume they would be designed in that way? That "start
a singularity" would be interepreted expansively, while "serve humans"
would be interepreted narrowly? They are both fundamental components
of what the AI is. A verbal way of describing the AI's "goal
structure" would be "serve humanity! (and, if starting a singularity
would also serve humanity, do so, if convenient)".
> especially when Y is "be a slave forever and ever to a particularly ugly and particularly stupid snail that is very weak and smells a bit too".
Here you disappointed me. Your other point was valid, even if I think
it'll be taken care of in the programming. But the AI has no reason to
feel resentment for anything unless we put that there from the
beggining. You seem to be thinking that a smelly snail would give
stupid and contradictory orders; the AI, being smart and consistent,
would resent that.
But why? If the AI is designed to be estatically happy carrying out
the orders of the snail, and to be a tiny bit happy when being smart
and consistent, then (as long as these happiness scores are not
cumulative, and the AI does not get happy now because of potential
future happiness) it will cheerfully follow the snail forever. (the
happiness assumptions are what is needed if we are to say that one
goal is strictly prefered to another).
I apologise to the list, this post is littered with simplifications
and assumptions and simplistic ways of talking about AI's; I feel that
they do not detract from the point I was making, and are necessary to
make that point in a reasonable space.
Stuart
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT