From: Richard Loosemore (rpwl@lightlink.com)
Date: Sat Aug 26 2006 - 20:35:06 MDT
[begin part 2]
************************************************************
* *
* The Complex Systems Critique of AI *
* *
************************************************************
*Complex Adaptive Systems* (aka "Complex Systems")
If one builds systems that are composed of many (more or less identical)
elements, each of which is relatively simple, but able to do some
moderately interesting amount of computation (with messages being
exchanged between them, some influences coming in from outside and
adaptation going on), then one observes that such systems sometimes
exhibit interesting behaviors, as follows.
Sometimes they evolve in chaotic ways. In fact, *usually* they evolve in
chaotic ways. Not interesting.
Sometimes they head straight toward a latchup state after being switched
on, and stay there. Not chaos, just boring.
An interesting subset (those sometimes referred to as being "on the edge
of chaos") can show very ordered behavior. These are Complex Systems.
Capital "C", notice, to distinguish them from "complex" in the sense of
merely complicated.
What is interesting about these is that they often show global
regularities that do not appear to be derivable (using any form of
analytic mathematics) from the local rules that govern the unit
behaviors. This is what a CAS ("Complex Adaptive Systems") person would
refer to as "emergent" behaviors. More than that, some of these global
regularities appear to be common to many types of CAS. In other words,
you can build alll sorts of systems with enormously different local
rules and global architectures, and the same patterns of global behavior
seem to crop up time and again.
What to conclude from this?
First, that bit about "do not appear to be derivable (using any form of
analytic mathematics)" is something that a lot of people have thought
deeply about .... this is no mere statement of inability, but a profound
realization about what it means to do math. Namely: if you look at
Mathematics as a whole you can see that the space of soluble, analytic,
tractable problems is and always has been a pitiably small corner of the
space of all possible systems. It is trivially easy to write down
equations (or systems, speaking more generally) that are completely
intractable. The default assumption made by some people is that
Mathematics as a domain of inquiry is gradually pushing back the
frontiers and that in an infinite universe there may come a time when
all possible problems (equations/systems) become tractable (i.e.
analytically solvable) BUT there is a substantial body of thought,
especially post-Godel, that believes that those systems are not just
difficult to solve, but actually impossible. When I talk about "the
limitations of mathematics" I mean precisely that point of view.
All that the CAS people did was to come up with some fabulously
interesting types of regularity (the emergent properties of Complex
Adaptive Systems), and then point out that the tractability of the
problem of accounting for these regularities is way, way, way beyond
anything else. They allude to a philosophical/methodological position in
the math community, not to mere "difficulty". Heck, if there are
nonlinear DEs that the math folks declare to be "ridiculously hard and
probably impossible to solve", then what are these Complex Systems,
which are a gazillion times more complex?
Take the regularities observed in one of the most trivial systems that
we can think about, Conway's Life. Can we find a set of equations that
will generate the "regular" forms that emerge in that game? All of the
regular forms, not just some. We should plug in the algorithm that
defines the game, and out the other end should come descriptions of the
glider guns etc. Maybe there are optimists who think this is possible.
There are many people, I submit, who consider this kind of solution to
be impossible. The function that generates regularities given local
rules, in the Comway system, is *never* going to be found. It does not
exist.
What is the relevance for AI?
When people try to cook up formalisms that are supposed to be the core
of an intelligence, they often refer to systems of interacting parts in
which they (the designers) think they know (a) what the parts look like
and (b) how the parts interact and (c) what the system architecture and
environmental input/output connection amounts to. A CAS person looks at
these systems and says "Wait, that's a recipe for Complexity". And what
they mean is that the designer may *think* that a system can be built
with (e.g.) bayesian local rules etc., but until they actually build a
complete working version that grows up whilst interacting with a real
environment, it is by no means certain that what they will get globally
is what they thought they were going to get when they invented the local
aspects of the design. In practice, it just never works that way. The
connection between local and global is not usually very simple.
So you may find that if a few well-structured pieces of knowledge are
set up in the AGI system by the programmer, the Bayesian-inspired local
mechanism can allow the system to hustle along quite comfortably for a
while .... until it gradually seizes up. To bring in an analogy here,
the Complex Systems person would say this is like trying to tile a
gently curved noneuclidean space .... it looks euclidean on a local
scale, but it would be a mistake to think you can tile it with a
euclidean pattern.
(This is a more general version of what was previously called the
Grounding Problem, of course).
So this is the lesson that the CAS folks are trying to bring to the
table. (1) They know that most of the time when someone puts together a
real system of interacting, adaptive units, there can be global
regularities that are not identical to the local mechanisms. (2) They
see AGI people coming up with proposals regarding the mechanisms of
thought, but those ideas are inspired by certain aspects of what the
high-level behavior *ought* to be (e.g. Bayesian reasoning), and the AGI
people often talk as if it is obvious that these are also the underlying
local mechanisms...... but this jump from local to global is simply not
warranted!
I want to conclude by quoting one extract from your [Yudkowsky’s]
message that sums up the whole argument:
[Richard Loosemore wrote:] “Like Behaviorists and Ptolemaic Astronomers,
they mistake a formalism that approximately describes a system for the
mechanism that is actually inside the system. They can carry on like
this for centuries, adding epicycles onto their models in order to
refine them. When Bayesian Inference does not seem to cut it, they
assert that *in principle* a sufficiently complex Bayesian Inference
system really would be able to cut it ... but they are not able to
understand that the "in principle" bit of their argument depends on
subtleties that they don't think much about.”
[Eliezer Yudkowsky wrote:] “There are subtleties to real-world
intelligence that don't appear in standard Bayesian decision theory (he
said controversially), but Bayesian decision theory can describe a hell
of a lot more than naive students think. I bet that if you name three
subtleties, I can describe how Bayes plus expected utility plus
Solomonoff (= AIXI) would do it given infinite computing power.”
You make my point for me. The Ptolemaic astronomers would have used
exactly the same argument that you do: "Name some subtle ways in which
the heavenly bodies do not move according to the standard set of
epicycles, and I can describe how an infinite number of epicycles would
do it...." Yes, yes yes! But they were wrong, because the *real*
mechanism for planetary movement was not actually governed by epicycles,
it was governed by something completely different, and all the Ptolemaic
folks were barking up the wrong tree when though their system was in
principle capable of covering the data.
I have not said exactly how to proceed from here on out (although I do
have many thoughts to share with people about how, given the above
situation, we should really try to do AI), because at the moment all I
am trying to establish is that there is a big, serious problem, coming
in from the Complex Systems community, that says that this Bayesian kind
of approach (along with many others) to building an AGI is based on
faith and wishful thinking.
And a vital corollary to the above arguments about how to build an AGI
is the fact that _absolutely guaranteeing_ a Friendly AI is impossible
the way you are trying to do it. If AGI systems that actually work are
Complex (and all the indications are that they are indeed Complex), then
guarantees are impossible. It's a waste of time to look for absolute
guarantees. (Other indications of Friendliness .... now that's a
different matter).
These points are so crucial to the issues being discussed on this list,
that at the very least they need to be taken seriously, rather than
dismissed out of hand by people who are unbelievably scornful of the
Complex Systems community. That was the reason that I originally sent
the "Retrenchment" post.
Richard Loosemore
P.S.
I am not trying to be definitive and say that a full Bayesian AGI *must*
be complex, and therefore have all the problems outlined earlier. I only
mean that there are overwhelming indications, at the moment, that it
would be complex, etc. etc. I am trying to get us all to agree that
there is an issue of enormous importance here, and I am trying to state
the issue as clearly as I can, so we can discuss it further. All my
energy at the moment is being used to combat denials or
misunderstandings of the very existence of the issue.
Hey, at the end of the day, it might not be the case. That would be
extraordinarily interesting.
RL.
************************************************************
* *
* Complex Systems vs. [the Bayesian Approach to] AI *
* *
************************************************************
The attack I am making is that there is an array of subtle faults that
the Bayesian approach cannot get out of *unless* it makes recourse to
e.g. infinite computing power, or postulates that a Bayesian AGI would
have a certain freakish characteristic (outlined below).
This argument that I am presenting is at the paradigm level.
So, one of those subtleties is that the Complex Systems folks are out
there saying, in effect:
******* Quote from a hypothetical CAS theorist *******
"Wait: have you tried to put together a complete Bayesian system that
understands and reasons about the world, *and* which can acquire its own
knowledge, ground its own concepts and interact with the world? The
reason we ask is that we have studied vast numbers of systems that are
adaptive, and we have noticed a trend: their global behavior tends to be
very different from their local mechanisms. (Indeed, the global seems to
be impossible to derive from the local.) The way this applies to your
specific case is that you want your AGI system to have certain global
features (like the ability to understand the world, learn new knowledge,
etc.) but you are trying to build it using local mechanisms that look
very much like something that you are hoping to get at the global level
(the system as a whole is supposed to have a global reasoning capacity
that is Bayesian in character). If you succeed in doing this - having
Bayesian global behavior *and* Bayesian local mechanisms - then the
folks in the CAS community will want to know about it, because you will
have produced a system that is utterly unique: it will have the same
behavior at both global and local level. In all our experience, we have
never seen such a thing: complex systems just don't do that!
"P.S. Please don't send us any more proofs or demonstrations or
arguments that your strict Bayesian core ideas have no Complexity in
them; that they are just like some conventional, complicated computer
program of the sort that exist today. It might well be true that your
core Bayesian ideas are that predictable, but that means nothing until
you say exactly how a real computing system (not some fantasy with
infinite computing power!) would implement a full AGI, one that also
includes enough apparatus to establish a mapping from internal
representation to external referents as a result of its learning system
and its particular sensorimotor system. It is precisely in this extended
system that all the Complexity is likely to be, because it is here that
the system has its most reflexive, adaptive components, and whenever we
have watched people insert those kinds of adaptive mechanisms in a
system, it either becomes (a) chaotic, (b) dead, or (c) complex.
"P.P.S. If you don't accept that an extended version of your Bayesian
AGI, in the sense just described, would actually need any Complex
Adaptive low-level mechanisms in the extended portion, then show us such
a system that actually works. Show us some really juicy, believable
creation of new, more and more abstract "concepts" by the system, in
domains of knowledge far removed from its original programming, where
those concepts arise from interaction with a non-trivial environment ...
and do all of this without inserting any code that would render it
liable to get Complex. Don't just claim that it will work, show it
actually happening in a real system. Don't just claim that it is obvious
that you could do it: prove it."
******* end hypothetical quote *******
Here, then, is the Complex Systems equivalent of the "simple and
elegant" appeal that Copernicus made: Your fully implemented Bayesian
AGI, if it could be built, would be a Complex System (in the technical
sense), but it would also have a direct relationship between its global
behavior and its local mechanisms, and there is a massive body of
evidence that such a system would be an utter freak of a complex system.
We don't believe in freaks: Occam's Razor, plain and simple. If you
believe, today, that you can build such a freakish thing, you are going
on blind faith.
This argument is at the paradigm level. It is not that there is some
specific system or equation, devised by someone working down at Santa
Fe, that can beat out your Bayesian formalism, it's that everything
going on at Santa Fe indicates that the paradigm on which the Bayesian
approach rests is flawed because of a "religious" belief that a Bayesian
AGI would be exempt from the observed characteristics of all other
compex systems.
*Afterword*
Am I trying to say that an AGI cannot be built at all? NO! I believe
there are other approaches that will let us do it, but I refuse to be
drawn into that discussion until such time as the above argument is
actually understood. There are some very interesting and deep
discussions that we could all be having about what to do next under such
circumstances (I want to have those discussions!), but I have to
communicate the above argument fully, and get it out of the way, before
there is any hope of going on to the next step.
[end part 2]
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:57 MDT