From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Aug 03 2005 - 12:26:42 MDT
Daniel, I think at this point I have to say: Study some probability theory.
You're arguing, as though it were a basic principle of rationality, a
principle of probability assignment which simply would not be well-calibrated
in practice.
You seem to believe that in the absence of "specific support" - which is
apparently something you get to define, if none of the historically similar
situations from dogs to Lord Kelvin count as generalizable cases - you must
assign probability zero. This is flatly wrong. If you read
http://yudkowsky.net/bayes/technical.html you will see why you should never
assign probability zero to anything.
Since you cannot assign probability zero, what probability must you assign?
Note that I do not say "may" assign. Rationality is a precise art. All that
is not prohibited is mandatory, all that is not mandatory is prohibited.
Water does not choose where to flow but only flows downhill, and probability
flows similarly in a Bayesian reasoner.
Technical rules for assigning probability in the absence of specific support
include maximum entropy and Kolmogorov complexity. Neither is readily
applicable to calculate a quantitative probability for the AI case. The best
probability estimate we can get is by analogy to historical cases from dogs to
Lord Kelvin. End of story! You cannot demand a specific account of how an AI
could break free from a box, else assign probability zero. It is like asking
for a specific account of how an opponent could win a game of Go against you,
else assigning probability zero to your loss. It is like saying that a
lottery ticket has probability zero of winning, unless someone gives you a
specific reason why those numbers will win. You can (therefore must) assign
negligible non-zero probabilities to each element of the search space, and
assign a larger non-zero probability for the entire search space.
There is simply no principle of probability theory that says that, if you
cannot exhibit a specific element of the Go search space that wins against
you, you must assign probability zero to the whole search space without
searching it.
If you permit (and you must permit) historical generalizations about similar
but not identical situations, such as past games of Go, in the absence of
specific exhibited possible winning moves against you, then you must permit
historical generalizations about similar failures of physical theory and
failures of imagination, in the absence of specific exhibited possible winning
moves against you.
I think that's essentially the end of the discussion so far as I'm concerned.
You are simply using probability theory incorrectly. If you read up on
technical rules for assigning probability in the absence of specific support,
you will probably get a better idea of where your verbal argument goes wrong,
even though you cannot use these methods to calculate a quantitative
probability in this case.
Rationality is supposed to work. It is supposed to produce correct answers
and well-calibrated probability assignments. If you believe that some
principle of rationality requires you to assign a zero probability to
something that could actually go ahead and happen, or a negligible probability
to something that stands a good chance of really happening, then whatever you
are doing is not rationality. Do not say that your art failed you; you failed
your art.
I'm not sure there's anything anyone can say to you beyond that. You appear
to have leaped to a conclusion and to be using an alleged principle of
rationality to justify it, which principle accords not with probability
theory, nor exhibits qualitative correspondence to common sense. No one else
here agrees with your principle and you have made no case for it. If you
continue to appeal to the principle, you will convince yourself but no one else.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:23:00 MST