**From:** Ben Goertzel (*ben@goertzel.org*)

**Date:** Fri Jan 09 2004 - 06:19:44 MST

**Next message:**Metaqualia: "talk on foresight hp"**Previous message:**Tomaz Kristan: "Re: An essay I just wrote on the Singularity."**In reply to:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Next in thread:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Reply:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*> Ben Goertzel wrote:
*

*> >
*

*> > Eliezer,
*

*> >
*

*> > It might be significantly easier to engineer an AI with a 20% or 1%
*

*> > (say) chance of being Friendly, than to engineer one with a 99.99%
*

*> > chance of being Friendly. If this is the case, then the
*

*> > broad-physical-dispersal approach that I suggested makes sense.
*

*>
*

*> 1) I doubt that it is "significantly easier". To get a 1% chance you
*

*> must solve 99% of the problem, as 'twere. It is no different from trying
*

*> to build a machine with a 1% chance of being an internal combustion
*

*> engine, a program with a 1% chance of being a spreadsheet, or a document
*

*> with a 1% chance of being well-formed XML.
*

Actually, the analogies you're making are quite poor, because internal

combustion engines and spreadsheets and XML documents are not complex

self-organizing systems. With Friendly AI, we're talking about creating an

initial condition, letting some dynamics run (interactively with the

environment, which includes us), and then continually nudging the dynamics

to keep them running in a Friendly direction. This is a very different ---

and plausibly, much less deterministic -- process than building a relatively

static machine like a car engine or a spreadsheet.

It may well be that it's not possible to create a program with more than x%

chance of being Friendly, where x=20% or 40% or 93.555%, or whatever. I

don't have a strong intuition as to whether this IS or IS NOT the case...

*> 2) Ignoring (1), and supposing someone built an AI with a 1% real chance
*

*> of being Friendly, I exceedingly doubt its maker would have the skill to
*

*> calculate that as a quantitative probability.
*

Of course, that's true. But it's also true that, if someone built an AI

with a 99.999% chance of being Friendly, it's maker is not likely to have

the skill to calculate that as a quantitative probability. Making

quantitative predictions about this kind of system is next to impossible,

because the dynamic evolution of the system is going to depend on its

environment -- on human interactions with it, and so forth. So to make a

rigorous probability estimate you'd have to set various quantitative bounds

on various environmental conditions, human behaviors, etc. Very tricky ...

not just a math problem, for sure... (and the math problems involved are

formidable enough even without these environmental-modeling considerations!)

*> 3) So we are not talking about a quantitative calculation that a program
*

*> will be Friendly, but rather an application of the Principle of
*

*> Indifference to surface outcomes. The maker just doesn't really know
*

*> whether the program will be Friendly or not, and so pulls a probability
*

*> out of his ass.
*

There are a lot of intermediate cases between a fully rigorous quantitative

calculation, and a purely nonrigorous "ass number."

After all, should you ever come up with a design that you think will ensure

Friendliness, you're not likely to have a fully rigorous mathematical proof

that it will do so ... there will be a certain amount of informal reasoning

required to follow your arguments.

*> 4) Extremely extensive research shows that "probabilities" which people
*

*> pull out of their asses (as opposed to being able to calculate them
*

*> quantitatively) are not calibrated, that is, they bear essentially no
*

*> relation to reality.
*

Sure, but your statement doesn't hold when applied to teams of scientists

making careful estimates of probabilities events based on a combination of

science, math and intuition. In this case, estimates are still imperfect --

and errors happen -- but things are not as bad as you're alleging.

*> 6) And finally, of course, the probabilities are not
*

*> independent! If the
*

*> best AI you can make isn't good enough, a million copies of it don't have
*

*> independent chances of success.
*

This depends a great deal on the nature of the AI.

If you're creating an AI that is a dynamical system, and you're building an

initial condition and letting it evolve in a way that is driven partially by

environmental influences, then if you run many copies of it independently

a) of course, the dynamics of the different instances are not

probabilistically independent

b) nevertheless, they may be VERY far from identical, and may come to a wide

variety of different outcomes

My dreamed idea (which wasn't so serious, by the way!) did not rely upon the

assumption of complete probabilistic independence between multiple evolving

instances of the AI. It did rely upon the idea of a self-modifying AI as a

pragmatically-nondeterministic, environmentally-coupled system rather than a

strongly-deterministic system like an auto engine.

*> What's wrong with this picture:
*

*>
*

*> a) Confusing plausibility with frequency;
*

*> b) Assigning something called a "probability" in the absence of a theory
*

*> powerful enough to calculate it quantitatively;
*

*> c) Treating highly correlated probabilities as independent;
*

*> d) Applying the Principle of Indifference to surface outcomes
*

*> rather than
*

*> elementary interchangeable events; and
*

*> e) Attempting to trade off not knowing how to solve a problem for
*

*> confessing a "1%" probability of success.
*

Sorry, I am definitely not guilty of errors a, c or d

As for error b, I don't think it's bad to call an unknown quantity a

probability just because I currently lack the evidence to calculate the

value of the quantity. b is not an error.

As for e, there may be some problems for which there are no

guaranteed-successful solutions, only solutions that have a reasonable

probability of success. You seem highly certain that Friendly AI does not

lie in this class, but you have given no evidence in favor of your

assertion.

*> And if you're wondering why I'm so down on this, it's because it seems to
*

*> me like yet another excuse for not knowing how to build a Friendly AI.
*

Actually, I *do* know how to build a Friendly AI.... ;-)

[I wasn't so sure a year ago, but recent simplifications in the Novamente

design (removing some of the harder-to-control components and replacing them

with probabilistic-inference-based alternatives) have made me more

confident.]

But I can't *guarantee* this AI will be Friendly no matter what; all I can

give are reasonable intuitive arguments why it will be Friendly. The

probability of the Friendliness outcome is not easy to calculate, as you've

pointed out so loquaciously.

And then I wonder whether that my "reasonable intuitive arguments" ---- and

*all* human arguments, whether we consider them rigorous or not; even our

best mathematical proofs --- kinda fall apart and reveal limitations when we

get into the domain of vastly transhuman intelligence. So I tend to apply

(a mix of rigorous and intuitive) probabilistic thinking when thinking about

transhuman AI's that are just a bit smarter than humans ... and then rely on

ignorance-based Principle of Indifference type thinking, when thinking about

transhuman AI's that are vastly, vastly smarter than any of us.

-- Ben G

**Next message:**Metaqualia: "talk on foresight hp"**Previous message:**Tomaz Kristan: "Re: An essay I just wrote on the Singularity."**In reply to:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Next in thread:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Reply:**Eliezer S. Yudkowsky: "Re: Dispersing AIs throughout the universe"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:43 MDT
*