From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Wed Jan 19 2005 - 13:24:54 MST
Brett Paatsch wrote:
>
> "Imagine that you wake up one morning and your left arm
> has been replaced by a blue tentacle. The blue tentacle
> obeys your motor commands - you can use it to pick up
> glasses, drive a car, etc. How would you explain this
> hypothetical scenario? Take a moment to ponder this
> puzzle before continuing."
>
> So I did imagine it. I imagined it in good faith, and I imagined it
> consistent with a spirit of exploration and good will built that Eliezer
> had established through the early part of his essay.
What about the spirit of cunning plots and mischief? For the life of me, I
don't understand why people are so shocked to find that I have a goofy
sense of humor; it's not as if my writings don't provide ample warning.
Anyway...
I'm trying to forge rationality into a new and more coherent art, reusing
my l33t build-a-mind-from-scratch skillz to go beyond that accumulated
handicraft of rationality passed down from generation to generation.
Sometimes I may say things that shock a listener raised in the ancient
tradition.
> Where Eliezer had placed "spoiler space", I stopped reading and I
> wrote down my explanation. (I'd been reading with pen in hand and
> making critical notes in the margin.)
Got a scanner? I wouldn't mind seeing a copy of the notes (if that
wouldn't be too much effort for you).
> It seemed to be fair and
> scientific to provide an answer *before* reading on so as not to
> contaminate the experiment. )
>
> I wrote (and I quote):
>
> "I'd "explain" it provisionally as some surprising organisation
> of people had entered my house and replaced my arm whilst
> I slept with technology I didn't know existed.
>
> I'd be bewildered. Frightened even. But I'd not think "magic"
> had occurred".
I also wouldn't think "magic" had occurred. The magic hypothesis is rather
vague without further specification, has tended to be disproved to the
extent it has been historically specified, and makes no particular
prediction of my arm being replaced with a tentacle.
> And then, with the heightened curiosity of one who has escalated
> their commitment I went back to see what Eliezer the Bayesian,
> Eliezer the spreader-of-analogical-probability-clay-mass would have
> done.
>
> And he'd written this.
>
> "How would I explain the event of my left arm being replaced
> by a blue tentacle? The answer is that I wouldn't. It isn't going
> to happen."
>
> Email perhaps can't convey my exact reaction to that but here's
> the comments I wrote in the margin.
>
> ----
>
> "No. No. No. You cheated Eliezer. You cheated!
>
> You can't assign a probability of zero!*
>
> Not fair!! You said it did happen. You're being dishonest with the
> data to say it's not going to happen."
Yep, that's a traditional response. :P
I was hoping for pretty much that reaction from my readers, who I assumed
would be traditional rationalists.
The ideal of traditional rationality is that reality is allowed to tell you
anything it wants, and you ought to shut up and listen - a stance arising
from the sad human tendency to deny experimental evidence when it conflicts
with something more valuable, like hope or authority.
What's less commonly appreciated is that reality does *not* tell you just
anything. Reality is *extremely constricted* in what it tells you, far
more constricted than human storytelling. Truth is *not* stranger than
fiction; humans just have a warped idea of what constitutes normality. The
theory of conservation of momentum is not that momentum is conserved *most*
of the time or *nearly all* of the time, it is that momentum is conserved
*every single time*. It isn't a coincidence, we hypothesize, that
conservation of momentum has been observed on every occasion thus yet where
humanity has had opportunity to test it; it's because reality obeys this
*absolute* rule, and this absolute rule has given rise to humanity's
experience so far.
Fallible humans may not dare to assign a probability of unity to the
probability of conservation of momentum. Let us say we assign a
probability of at most 90%, meaning that if we took ten contemporary
physical hypotheses of equal status, we would expect at least one to be
disconfirmed a millennium hence. But the hypothesis of conservation of
momentum is not that momentum is conserved 90% of the time or even 99.9999%
of the time. The hypothesis of conservation of momentum is that momentum
is conserved 100.00000% of the time. We may be uncertain, but the
hypothesis of "conservation of momentum" hypothesizes a state of affairs in
which reality is *not* uncertain; a reality in which it is *absolutely
certain* that momentum will be conserved on each and every occasion.
The absolute character of physical law is foreign to human thinking. When
humans make rules, there's always some unwritten give-and-take, some room
for compromise. Even some atheists are shocked by the uncompromising
attitude of radical foaming-at-the-mouth atheist hard-liners like myself,
that there is *no* room for magic in the universe, that there is *no*
separate magisterium reserved for thoughts "spiritual" or "religious" or
"magical". And to those who claim that every belief system partakes of the
transcedent, I say: Yes, every known culture divides the universe into a
mundane realm of verifiable assertions and a sacred realm where sloppy
thinking is allowed; but I'm saying that's a mistake.
It is not a coincidence that every sacred hypothesis in human history which
infringes on testability has been falsified. (And that is nearly all
sacred hypotheses. The makers of such hypotheses lack the skill in
rationality to make the hypotheses genuinely unfalsifiable. If they
understood what constitutes Bayesian evidence, they wouldn't be
constructing sacred hypotheses.) The most probable cause of humanity's
experience so far is that we live in an *absolutely* mundane universe,
mundane in every place and every time with 100.00000000000000% frequency.
By asserting that conservation of momentum is *absolute*, the hypothesis
sticks out its neck very far, and perhaps its neck will be sliced off by an
experiment that violates conservation of momentum - but that's part of the
point. If someone reports an experiment that violates conservation of
momentum, you shouldn't chalk it down to a rare exception to the general
rule (maybe someone negotiated the laws of physics down a little from their
extreme and unreasonable position that momentum should be conserved on
every single occasion).
Experiments in cognitive psychology show that people fear air travel more
than car travel. Why? Because airplane accidents are more frequently
reported in the media. People's fear of a particular hazard is, generally,
directly proportional to how often that hazard is reported in the media.
Rare hazards that kill lots of people, like airplane crashes, tend to be
reported in the media often; common small hazards go relatively unnoticed.
This bias is only one of many experiments where people have been observed
reason, not from the actual statistical frequencies of cases, but from the
most attention-grabbing cases or most commonly reported cases. I think
that part of the Way is trying to fit yourself to the real world, to the
actual statistical frequency of events - deliberately thinking about the
statistical likelihood of any sample case presented for your attention.
No one in all human history has ever woken up with a functioning tentacle
in place of their arm. You should have noticed that when I asked you to
find an explanation for it. Yes, you were tricked. But it's a kind of
trickery that goes on all the time and which I wish my readers to notice.
For example, a thought experiment which postulates "zombies", people that
behave exactly like people in this universe except that they are not
conscious. (Why do they write papers on consciousness, including
discussion of zombie experiments, if they are not conscious? For more on
this see Dennett's "The Unimaginable Preposterousness of Zombies".) Or
consider Damien Broderick's swift reaction to John C. Wright's attempt to
quote Scrooge as an argument against skepticism. Damien didn't actually
use my phrase "logical fallacy of generalization from fictional evidence",
but I hope my writing had something to do with it.
Occasionally I tread on the futile task of trying to persuade people not to
buy lottery tickets, and they say something along the line of "Someone has
to win!" or "You can't win if you don't play!" To which the answer is,
"'Someone' will not be you. You will not win the lottery, period. I could
make a hundred thousand statements of equal strength and not be wrong even
once. I have *godlike* confidence that you will not win the lottery. Stop
thinking about the pleasantness of the outcome, stop attending to this
improbable event, because it will *not happen*." I ask people to stop
buying lottery tickets, and they think, "But what if I would have won?",
and imagine a thought experiment which, though physically possible - not
*absolutely* impossible like violating conservation of momentum - has a
probability so low that *they shouldn't be thinking about it while making
the decision*.
I would advise philosophers who indulge in thought experiments to
consciously weight the emotional force of a thought experiment by the
probability of that thought experiment, so that they may learn to live in
this, our real world.
Lottery players are led astray by attending to hypothetical events of great
emotional intensity and extremely low probability. So too, John C. Wright
was led astray by attending to a powerful and sympathetic experience,
Scrooge's discovery of magic and the triumphant comeuppance of skepticism,
which never actually happened. Nor will it happen, ever, in this our
mundane universe.
You won't wake up with a blue tentacle in place of an arm. Remember that,
the next time someone asks you to imagine the impossible, or even the
statistically improbable. You followed where I asked you to go, imagined
what I asked you to imagine, and then found yourself betrayed. But next
time you will notice when someone tries to lead you astray.
Let us learn to live in this universe the way it really is, attending to
the real frequency of events instead of the frequency of media reports of
events. When someone asks us to imagine a magical outcome, let us forget
all of the novels we have read, and all the movies we have seen, and all
the hopes of our childhood, and remember that the observed frequency is zero.
At the same time, let's not forget how ridiculous the 20th century would
have sounded if you'd reported it to a 19th-century listener. But reality
is very constrained in what kind of ridiculousness it presents us with.
Not one of the ridiculous things that happened in the 20th century violated
conservation of momentum.
Oh... and if you *do* wake up with a tentacle in place of an arm... it's
probably not because anyone snuck into your room; there must be a simpler,
more likely explanation you didn't think of, or the event wouldn't have
happened. When you have only a poor explanation, one that doesn't make
things ordinary in retrospect, just admit you don't have an explanation,
and keep going. Poor explanations very, very rarely turn out to be
actually correct. A gang of people sneaking into your room with unknown
technology is a poor explanation. Whatever the real explanation was, it
wouldn't be that. If that's the best you can do, then "I wasn't expecting
this and I have no clue why it happened or what will happen next" is a far
superior answer. If I demanded that you produce a specific hypothesis
anyway, you should have told me to stuff it. Why develop a habit of
producing hypotheses that can't be right? What good will it serve you?
In real life, sometimes we don't know what happened. Real life is where we
will actually apply our l33t rationality skillz, so "I don't know" was the
correct answer, taking the impossible experience as a fixed given. If I'd
sworn that I really did possess some concrete reason to anticipate that you
might *actually* wake up with a tentacle, and asked you to guess my good
reason, "I don't know" would have been the correct answer (taking my
rationality as a fixed given).
> Most of what I know of Bayesian reasoning I know as a result of
> reading Eliezer's two essays on it.
If you enjoy the art, read more literature on it. "Rational Choice in an
Uncertain World" is good. If you're ready for academic stuff, start with
"Judgment Under Uncertainty".
> So perhaps if my understanding
> of Bayesian reasoning or inference is wrong I can escape by
> blaming Eliezer for it :-)
Whoa, I'm a highly atypical Bayesian. If you're judging Bayesian academia
by me, that's, er, not a good idea. I did warn my readers that Technical
Explanation would be controversial.
> I suspect, on the basis of those two essays that I am a Bayesian
> although I didn't know I was and so I haven't been calling myself
> one. The merit I see in the Bayesian approach is that it manages
> uncertainties more carefully and consistently then most people do
> intuitively. [And boy does the world need that].
Human beings aren't designed as Bayesians (though our thoughts sometimes
have Bayes-structure, which is why we work at all). The word "Bayesian"
usually refers, not to someone who actually implements Bayesian principles
(for that is humanly unattainable), but to someone who espouses Bayesian
principles. I don't think you could be a Bayesian without knowing it,
unless you had unwittingly demanded that people be principled about
assigning prior probabilities, or some such stance which today is commonly
known as "Bayesian".
> So, it's 2005. I'm a Bayesian. And as long as I'm wearing metaphorical
> teeshirts I'm also a Bright.
>
> Regards,
> Brett Paatsch
>
> * Eliezer's assigning a probability of zero to observed facts however
> unlikely those facts might have been a priori is the reason for my
> heading this post Bad Bayesian - no buscuit.
You can't assign a probability of 0.000000..., but you could theoretically
assign a probability of 1e-100 if you think you can get away with it,
because it isn't actually *zero* - the logarithm is minus a thousand
decibels, but not actually negative infinity. 1e-100 is effectively zero,
but there's a world of difference between effectively zero and actually
zero. My point is that you should be wary of probabilities which are,
*given* the dominant physical hypothesis, actually zero or effectively zero.
If your explanation: "A secret organization of people entered my house and
replaced my arm with a tentacle using unknown technology" doesn't make you
anticipate (even just a little) waking up with a tentacle tomorrow in this
our real world, then it's a poor explanation. For it is this, our real
world, in which you must live. Nor should you bother trying to develop a
better explanation. For in this, our real world, you will have no need of it.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:50 MDT