From: Michael Wilson (mwdestinystar@yahoo.co.uk)
Date: Tue Sep 20 2005 - 04:08:11 MDT
Ben Goertzel wrote:
> An unpredictable emergent phenomenon in a system is a behavior in a whole
> system that we know can in principle be predicted from the behavior of the
> parts of the system -- but carrying out this prediction in practice is
> extremely computationally difficult.
I'm not going to continue criticising the 'we must use Complexity theory'
position, as I think that debate is past the point of diminishing returns.
However given the potential for confusion in the extended back-and-forth
I think I should clarify my (and to a lesser extent, the SIAI's) position.
1. The requirement for a certain amount of strong predictability comes from
the need for Friendliness, analysis that suggests that unless you strongly
constrain goal system evolution it will be highly unpredictable, and the
simple fact that when humans are confident something will work, without
having a technical argument for why it will work, we're usually wrong.
2. Thus the SIAI has the design requirement; goal system trajectory must
reliably stay within certain bounds, which is to say that the optimisation
targets of the overall optimisation process must not drift out of a
certain region. This is a very specific and limited kind of predictability;
we don't need specific AI behaviour or cognitive content. I agree that the
task would be impossible if one were trying to predict much more than just
the optimisation targets. I am happy to have all kinds of emergence and
Complexity occuring as long as they stay within the overall constraints,
though theory and limited experimental experience suggests to me that there
will be a lot less of this than most people would expect.
3. If that turns out to be impossible, then we'd agree that AGI development
should just go ahead using the best probabilistic methods available (maybe;
it might make sense to develop IA first in that case). But we shouldn't
write something this important off as impossible without trying really
hard first, and I think that many people are far too quick to dismiss this
so that they can get on with the 'fun stuff' i.e. actual AGI design.
4. Various researchers including Eliezer have spent a fair amount of time
on this, and so far it looks probable that it is possible given arbitrary
AGI design that have access to unbounded computing power. The critical
question is whether there is a tractable design for an AGI that satisfies
the structural requirements of these theories. This is something that I'm
working on; unfortunately I'm not aware of anyone else working on it at
present, though I certainly wish there was.
5. Any system compatible with the known approaches to strong verification
of Friendliness will need to be consistently rational, which is to say
Bayesian from the ground up and have the structural property of being
'causally clean', although not necessarily driven by expected utility.
When I first accepted these constraints, they seemed onerous to the point
of making a tractabale architecture impossible; all the 'powerful'
techniques I knew of (improved GAs, stochastic codelets, dynamic-toplogy
NNs, agent systems etc) were thoroughly probabilistic* and hence difficult
to use or completely unusable. But after a period of research I now
believe that there are acceptable and even superior replacements for all
of these that are compatible with strong verification of Friendliness.
I'm not going to defend that as anything more than a personal opinion at
this time.
* Annoying terminology conflict; 'probabilistic methods' are not the same
thing as 'probabilistic logic'. The former are problem-solving techniques
that don't reliably obey constraints and/or fail to show a reliable
minimum performance in relation to normative decision theory; an analogy
could be drawn to 'soft real time' instead of 'hard real time'. This is
why saying 'Bayesian logic' to mean 'probabilistic logic' is not too bad
an idea even if it causes people to fixate on one particular derrivation.
6. Basically, a rational system of this kind avoids unwanted interactions
that would violate top-down constraints by constraining the way in which
components can inteact as you string them together. The resulting
structure could reasonably be called fractal; combining any set of
rational components in a rational framework produces a combined system
that is still rational. Yes, I mean something specific and moderately
complicated by 'rational' which I don't have space to fully describe.
Yes, doing this without sacraficing tractability is hard, but at present
I am optimistic that it will not turn out to be impossible. Yes, I am
working on a practical experiment/demonstration, this will take some
time, and I wish I had more resources to do it.
7. Note that this introduces the notion of 'kinds of Complexity'; a
system of this kind would be 'Complex' in some respects and non-Complex
in others. There are plenty of existing technological systems that
already look like this, so I see no reason to object to it.
8. Neither I nor the SIAI have claimed that this is the only way to build
AGI; in fact if it was we'd sleep a lot safer at night. Unfortunately it
seems entirely possible to build an AI using 'emergence', given enough
brute force, neuroscience and/or luck. The SIAI's claim is that this is
a /really bad idea/, because the result is highly likely to be iminical
to human goals and morals. The claims that any transhuman intelligence
will renormalise to a rational basis, and that this is actually a better
way to develop AGI regardless of Friendliness concerns, are weaker ones
and again stand only as opinion in public at this time.
9. No-one associated with the SIAI denies that the brain is an example
of a 'Complex system', or that emergence as a concept won't be useful
for studying it. We do claim that it is a horrible mess from and that it
isn't terribly relevant to the task of building an AGI compatible with
strong Friendliness verification. The position that closely mimicking
the brain isn't a good way to build AGI regardless of Friendliness is
again opinion, but the position that an AGI built in this fashion will
probably be Unfriendly is strongly justified from the previously
mentioned arguments.
10. The issue of 'Friendliness content' is genuinely seperate from
'Friendliness structure' and hence 'strong Friendliness verification'.
The latter is perhaps a misnomer, as there is some theory that is
applicable to any attempt to verify that an RPOP will do something
specific, though it is true that there are some things we would probably
want an FAI to do that require additional theory to describe and verify.
Arguments about whether CV, or 'joy, choice and growth', or domain
protection, or hedonism or Sysops or anything similar are a good idea
are debates about Friendliness content. This is important, but it's
well seperated from issues of structural verification and tractable
implementation, and different in character (because it involves what
we want instead of how to do it).
11. Personally I am quite skeptical of Eliezer's ideas about
Friendliness content, but I support his very important and (as far as
I can see) valid work on structural verification. I do wish he'd
publish more, but that criticism can be levelled against most people
working on AGI, including me. It's true that neither Eliezer nor the
SIAI has done much work on tractability, which is the main reason
why I'm working on it. However I agree that the question of how to
build something in the real world should follow that of how to build
it in principle, and that people need to be convinced about the
desireability and theoretical possibility of structural verification
(of AGIs as general optimisers) before it makes sense to argue about
if we can do it with real software on contemporary hardware.
12. Finally, my objection to claims about the value of Complexity theory
were summed up by one critic's comment that "Wolfram's 'A New Kind of
Science' would have been fine if it had been called 'Fun With Graph
Paper'". The field has produced a vast amount of hype, a small amount
of interesting maths and very few useful predictive theories in other
domains. Its proponents are quick to claim that their ideas apply to
virtually everything, when in practice they seem to have been actually
useful in rather few cases. This opinion is based on coverage in the
science press and would be easy to change via evidence, but to date
no-one has responded to Eliezer's challenge with real examples of
complexity theory doing something useful. That said, general opinions
such as this are a side issue; the specifics of AGI are the important
part.
> It may be that intelligence given limited resources intrinsically
> requires stochastic algorithms, but that is a whole other issue.
> Stochastic algorithms are not all that closely related to emergent
> phenomena -- one can get both emergence and non-emergence from both
> stochastic and non-stochastic algorithms.
I agree, but in practice it does seem that stochastic systems are more
likely to show/use emergence and vice versa.
That said, I really must stop spending so much time writing emails.
* Michael Wilson
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:52 MDT