**From:** Eliezer S. Yudkowsky (*sentience@pobox.com*)

**Date:** Thu Jan 08 2004 - 19:55:08 MST

**Next message:**Mitchell Porter: "Re: Recursive self-improvement curves"**Previous message:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**In reply to:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**Next in thread:**Metaqualia: "Re: Dispersing AIs throughout the universe"**Reply:**Metaqualia: "Re: Dispersing AIs throughout the universe"**Reply:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Ben Goertzel wrote:

*>
*

*> Eliezer,
*

*>
*

*> It might be significantly easier to engineer an AI with a 20% or 1%
*

*> (say) chance of being Friendly, than to engineer one with a 99.99%
*

*> chance of being Friendly. If this is the case, then the
*

*> broad-physical-dispersal approach that I suggested makes sense.
*

1) I doubt that it is "significantly easier". To get a 1% chance you

must solve 99% of the problem, as 'twere. It is no different from trying

to build a machine with a 1% chance of being an internal combustion

engine, a program with a 1% chance of being a spreadsheet, or a document

with a 1% chance of being well-formed XML.

2) Ignoring (1), and supposing someone built an AI with a 1% real chance

of being Friendly, I exceedingly doubt its maker would have the skill to

calculate that as a quantitative probability. To go over a program

specification and correctly, quantitatively calculate the probability that

it meets a certain criterion requires knowing exactly what that criterion

is, which variables the process flow critically depends on, and how those

variables contribute to the final probability, and so on. To correctly

calculate that a poorly assembled program (one representing the limit of

its maker's skill) has a 1% chance of being Friendly - even to within an

order of magnitude! - requires a skill level considerably, no, enormously

higher than that required to build a program with a 99.99% chance of being

Friendly; you must have reduced the entire problem to math, know the exact

criterion of success, tracked the dependency of success on every one of

the variables, and be capable of performing this calculation as

quantitative math for programs poorly assembled. NASA successfully

designed, built, and launched multiple space shuttles on multiple

successful missions, but their attempt to calculate a quantitative

probability of mission failure was statistically laughable and

demonstrably incorrect. If someone were to build, at the limit of their

skill, a program with a 1% real chance of being Friendly, and an observer

correctly calculated this probability, the observer would have to be a god.

3) So we are not talking about a quantitative calculation that a program

will be Friendly, but rather an application of the Principle of

Indifference to surface outcomes. The maker just doesn't really know

whether the program will be Friendly or not, and so pulls a probability

out of his ass. This reminds me of the story about when Australia was

starting its national lottery, and the television crews interviewed a man

in the street, asking him what he thought his chances were of winning.

"50/50", he said, "either I win or I don't."

4) Extremely extensive research shows that "probabilities" which people

pull out of their asses (as opposed to being able to calculate them

quantitatively) are not calibrated, that is, they bear essentially no

relation to reality. People use the term "99% probable" as a sort of

emotional ejaculation indicating that they believe in something really

strongly. There is no relation to actual probabilities. On empirical

tests, the range of surprises for 98% confidence intervals (where the

person gives an upper limit of 99% confidence and a lower limit of 99%

confidence) ranges between 30% and 60%, and the most common number I have

seen is 45%. This incredible overconfidence and enormously poor

calibration gets worse as the difficulty of the problem increases. The

problem is, of course, is that "probabilities" that people pull out of

their asses are just not related to reality in any way, nor do most people

realize that probabilities need to be calibrated. For example, someone

who says that the chance of an AI undergoing a hard takeoff is "a million

to one" is implying that he could be asked a million questions of equal

difficulty and expect to get at most one of them wrong. In reality, if

you cannot do the calculation and get a quantitative probability, you DO

NOT KNOW the probability and that is all there is to it. Research also

shows that this is one of the "resistant delusions" - most people,

confronted with the research that shows that making up probabilities

doesn't work, go on making up probabilities; being told about the research

fails to have the emotional impact that would be needed to overcome the

fun of making up probabilities. This is why I am trying to nag everyone

in the transhumanist community into reading Tversky and Kahneman's

"Judgment under uncertainty."

5) Plausibility is not the same as frequency. If you evaluate the

evidence (hopefully Bayesian evidence) back and forth and wind up by

estimating a 95% probability that "2 + 2 = 4", it doesn't mean you think

"2 + 2 = 4" on 19 out of 20 occasions.

6) And finally, of course, the probabilities are not independent! If the

best AI you can make isn't good enough, a million copies of it don't have

independent chances of success.

So to sum up, what we have is something like this. A person - let us

suppose for the sake of tradition that it is an Australian man - buys a

lottery ticket. On the surface, the odds seem like they should be 50/50,

since either he wins or he doesn't. It seems, though, that a lot of

people think it's really unlikely that he'll win the lottery, and so he

concedes that the probability might be a little lower. Checking how

strongly he feels about it, he finds that he wants to believe but is

afraid of being proven wrong, a state of mind that he describes with the

phrase "20% probability", which allows a satisfying hope of winning, while

still being low enough to ward off most disappointment in the event of

failure. However, the person comes up with a clever plan. Suppose that

he puts the lottery ticket under a hat. Then, once he knows the winning

number, he'll pull the hat off the lottery ticket. If the ticket doesn't

win the first time, he'll put the hat back on, and pull it off again.

Without knowing the winning numbers (at the present time), he calculates

in advance that on each occasion his chance of seeing the winning number

under the hat is 20%. So if he does this just 20 times, his chance of

winning is 99%!

What's wrong with this picture:

a) Confusing plausibility with frequency;

b) Assigning something called a "probability" in the absence of a theory

powerful enough to calculate it quantitatively;

c) Treating highly correlated probabilities as independent;

d) Applying the Principle of Indifference to surface outcomes rather than

elementary interchangeable events; and

e) Attempting to trade off not knowing how to solve a problem for

confessing a "1%" probability of success.

And if you're wondering why I'm so down on this, it's because it seems to

me like yet another excuse for not knowing how to build a Friendly AI.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Mitchell Porter: "Re: Recursive self-improvement curves"**Previous message:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**In reply to:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**Next in thread:**Metaqualia: "Re: Dispersing AIs throughout the universe"**Reply:**Metaqualia: "Re: Dispersing AIs throughout the universe"**Reply:**Ben Goertzel: "RE: Dispersing AIs throughout the universe"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:43 MDT
*