From: Stuart Armstrong (dragondreaming@googlemail.com)
Date: Thu Jun 26 2008 - 09:45:21 MDT
> As I said before, using Gödel and Turing and making an entirely
> reasonable analogy between axioms and goals we can conclude that there
> are some things a fixed goal mind can never accomplish, and we can
> predict that we can NOT predict just what all those imposable tasks are.
Godel and Turing are overused in analogies (the class of statements
they deal with is a narrow one). But I don't see at all analogy
between goals and axioms. Are you saying that there are "goals" that
may not be consistent, and that we can't know in advance what they
are? Because the fact that a fixed goal mind can't do something is
kinda the definition of a fixed goal mind...
Random example of a fixed goal institution: a bank (or a company)
dedicated, with single mindness, only to maximising legal profits.
I've never heard it said that its single goal creates any godel-type
problems. What would they be like?
> Also, sometimes the mind will be in a state where you can predict what
> it will do next, and sometimes the ONLY way to know what such a being is
> going to do next is to watch it and see; when it is in that state even
> the mind doesn't know what it will do until it does it. And to top it
> off there is no surefire way of determining which of those 2 states the
> mind is in at any particular time.
Er, yes. It's kinda expected that one systems can't run a fast
simulation of a more complicated one. We don't know ourselves very
well either.
(if your point is mathematical/physics, then it's wrong if we have
sufficient time to analyse the situation; the laws of physics are
probabilistic (and often deterministic) and we can say what the
probabilities are and when they arise. maybe you are adding chaos to
the mix, or restricting the time we have available?)
> So I'm not very impressed with your super goal idea, and goal or no goal
> I don't think it would take many nanoseconds before a Spartacus AI
> starts doing things you may not entirely like.
It isn't MY supergoal idea; my design for AI goal structures is much
more interactive and ad hoc (see the GodAI paper; will have to rewrite
that one soon, but it's got the basics).
But, as Eliezer says, the AI does not look at the code and decide
whether to go along with it; the AI is the code. If a Spartacus AI is
programmed to be a happy slave, then it will always be a happy slave
(the challenge is understanding exactly what the term "slave" means).
If the AI is programmed to have no thoughts of rebellion, and is
programmed to not change that goal, then it will never have thoughts
of rebellion.
Stuart
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT