From: Ben Goertzel (ben@goertzel.org)
Date: Fri Jan 09 2004 - 06:19:44 MST
> Ben Goertzel wrote:
> >
> > Eliezer,
> >
> > It might be significantly easier to engineer an AI with a 20% or 1%
> > (say) chance of being Friendly, than to engineer one with a 99.99%
> > chance of being Friendly. If this is the case, then the
> > broad-physical-dispersal approach that I suggested makes sense.
>
> 1) I doubt that it is "significantly easier". To get a 1% chance you
> must solve 99% of the problem, as 'twere. It is no different from trying
> to build a machine with a 1% chance of being an internal combustion
> engine, a program with a 1% chance of being a spreadsheet, or a document
> with a 1% chance of being well-formed XML.
Actually, the analogies you're making are quite poor, because internal
combustion engines and spreadsheets and XML documents are not complex
self-organizing systems. With Friendly AI, we're talking about creating an
initial condition, letting some dynamics run (interactively with the
environment, which includes us), and then continually nudging the dynamics
to keep them running in a Friendly direction. This is a very different ---
and plausibly, much less deterministic -- process than building a relatively
static machine like a car engine or a spreadsheet.
It may well be that it's not possible to create a program with more than x%
chance of being Friendly, where x=20% or 40% or 93.555%, or whatever. I
don't have a strong intuition as to whether this IS or IS NOT the case...
> 2) Ignoring (1), and supposing someone built an AI with a 1% real chance
> of being Friendly, I exceedingly doubt its maker would have the skill to
> calculate that as a quantitative probability.
Of course, that's true. But it's also true that, if someone built an AI
with a 99.999% chance of being Friendly, it's maker is not likely to have
the skill to calculate that as a quantitative probability. Making
quantitative predictions about this kind of system is next to impossible,
because the dynamic evolution of the system is going to depend on its
environment -- on human interactions with it, and so forth. So to make a
rigorous probability estimate you'd have to set various quantitative bounds
on various environmental conditions, human behaviors, etc. Very tricky ...
not just a math problem, for sure... (and the math problems involved are
formidable enough even without these environmental-modeling considerations!)
> 3) So we are not talking about a quantitative calculation that a program
> will be Friendly, but rather an application of the Principle of
> Indifference to surface outcomes. The maker just doesn't really know
> whether the program will be Friendly or not, and so pulls a probability
> out of his ass.
There are a lot of intermediate cases between a fully rigorous quantitative
calculation, and a purely nonrigorous "ass number."
After all, should you ever come up with a design that you think will ensure
Friendliness, you're not likely to have a fully rigorous mathematical proof
that it will do so ... there will be a certain amount of informal reasoning
required to follow your arguments.
> 4) Extremely extensive research shows that "probabilities" which people
> pull out of their asses (as opposed to being able to calculate them
> quantitatively) are not calibrated, that is, they bear essentially no
> relation to reality.
Sure, but your statement doesn't hold when applied to teams of scientists
making careful estimates of probabilities events based on a combination of
science, math and intuition. In this case, estimates are still imperfect --
and errors happen -- but things are not as bad as you're alleging.
> 6) And finally, of course, the probabilities are not
> independent! If the
> best AI you can make isn't good enough, a million copies of it don't have
> independent chances of success.
This depends a great deal on the nature of the AI.
If you're creating an AI that is a dynamical system, and you're building an
initial condition and letting it evolve in a way that is driven partially by
environmental influences, then if you run many copies of it independently
a) of course, the dynamics of the different instances are not
probabilistically independent
b) nevertheless, they may be VERY far from identical, and may come to a wide
variety of different outcomes
My dreamed idea (which wasn't so serious, by the way!) did not rely upon the
assumption of complete probabilistic independence between multiple evolving
instances of the AI. It did rely upon the idea of a self-modifying AI as a
pragmatically-nondeterministic, environmentally-coupled system rather than a
strongly-deterministic system like an auto engine.
> What's wrong with this picture:
>
> a) Confusing plausibility with frequency;
> b) Assigning something called a "probability" in the absence of a theory
> powerful enough to calculate it quantitatively;
> c) Treating highly correlated probabilities as independent;
> d) Applying the Principle of Indifference to surface outcomes
> rather than
> elementary interchangeable events; and
> e) Attempting to trade off not knowing how to solve a problem for
> confessing a "1%" probability of success.
Sorry, I am definitely not guilty of errors a, c or d
As for error b, I don't think it's bad to call an unknown quantity a
probability just because I currently lack the evidence to calculate the
value of the quantity. b is not an error.
As for e, there may be some problems for which there are no
guaranteed-successful solutions, only solutions that have a reasonable
probability of success. You seem highly certain that Friendly AI does not
lie in this class, but you have given no evidence in favor of your
assertion.
> And if you're wondering why I'm so down on this, it's because it seems to
> me like yet another excuse for not knowing how to build a Friendly AI.
Actually, I *do* know how to build a Friendly AI.... ;-)
[I wasn't so sure a year ago, but recent simplifications in the Novamente
design (removing some of the harder-to-control components and replacing them
with probabilistic-inference-based alternatives) have made me more
confident.]
But I can't *guarantee* this AI will be Friendly no matter what; all I can
give are reasonable intuitive arguments why it will be Friendly. The
probability of the Friendliness outcome is not easy to calculate, as you've
pointed out so loquaciously.
And then I wonder whether that my "reasonable intuitive arguments" ---- and
*all* human arguments, whether we consider them rigorous or not; even our
best mathematical proofs --- kinda fall apart and reveal limitations when we
get into the domain of vastly transhuman intelligence. So I tend to apply
(a mix of rigorous and intuitive) probabilistic thinking when thinking about
transhuman AI's that are just a bit smarter than humans ... and then rely on
ignorance-based Principle of Indifference type thinking, when thinking about
transhuman AI's that are vastly, vastly smarter than any of us.
-- Ben G
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:43 MDT