**From:** Eliezer S. Yudkowsky (*sentience@pobox.com*)

**Date:** Sat Mar 19 2005 - 23:25:06 MST

**Next message:**Mitchell Howe: "Marc Geddes Request"**Previous message:**Jeff Medina: "Reminder: SIAI Volunteers Meeting Tomorrow @ 3pm EST"**Maybe in reply to:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Next in thread:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Robin Hanson wrote:

*> At 12:57 AM 3/13/2005, Eliezer S. Yudkowsky wrote:
*

*>
*

*>> If I had to select out two points as most important, they would be:
*

*>> 1) Just because perfect Bayesians, or even certain formally imperfect
*

*>> Bayesians that are still not like humans, *will* always agree; it does
*

*>> not follow that a human rationalist can obtain a higher Bayesian score
*

*>> (truth value), or the maximal humanly feasible score, by deliberately
*

*>> *trying* to agree more with other humans, even other human rationalists.
*

*>> 2) Just because, if everyone agreed to do X without further argument
*

*>> or modification (where X is not agreeing to disagree), the average
*

*>> Bayesian score would increase relative to its current position, it
*

*>> does not follow that X is the *optimal* strategy.
*

*>
*

*> These points are stated very weakly, basically just inviting me to
*

*> *prove* my claims with mathematical precision. I may yet rise to that
*

*> challenge when I get more back into this.
*

We are at odds about what the math here actually *says*. I don't regard

the sequiturs (the points above where I say, "it does not follow") as

things that are a trivial distance from previously accomplished math.

They seem to me almost wholly unrelated to all work on Aumann Agreement

Theorems done so far.

Since state information is irrelevant to our dispute, it would seem that

we disagree about the results of a computation.

Here's at least one semisolid mathematical result that I scribbled down

in a couple of lines of calculus left to the reader, a result which I

intuitively expected to find, and which I would have found to be a much

more compelling argument in 2003. It is that, when two Bayesianitarians

disagree about probabilities P and ~P, they can always immediately

improve the expectation of the sum of their Bayesian scores by averaging

together their probability estimates for P and ~P, *regardless of the

real value*.

Let the average probability equal X. Let the individual pre-averaging

probabilities equal X+d and X-d. Let the actual objective frequency

equal P. The function:

f(d) = p*[log(x+d) + log(x-d)] + (1-p)*[log(1-x-d) + log(1-x+d)]

has a maximum at d=0, regardless of the value of p. f'(0)=0 and f''(0)

is negative. If my math hasn't misled me there's a couple of other

points where f'(d)=0 but they're presumably inflection points or minima

or some such. I didn't bother checking which is why I call this a

semisolid result.

Therefore, if two Bayesianitarian *altruists* find that they disagree,

and they have no better algorithm to resolve their disagreement, they

should immediately average together their probability estimates.

I would have found this argument compelling in 2003 because at that

time, I was thinking in terms of a "Categorical Imperative" foundation

for probability theory, i.e., a rule that, if all observers follow it,

will maximize their collective Bayesian score. I thought this solved

some anthropic problems, but I was mistaken, though it did shed light.

Never mind, long story.

To try and translate my problem with my former foundation without going

into a full-blown lecture on the Way: Suppose that a creationist comes

to me and is genuinely willing to update his belief in evolution from ~0

to .5, providing that I update my belief from ~1 to .5. This will

necessarily improve the expectation of the sum of our Bayesian scores.

But:

1) I can't just change my beliefs any time I please. I can't cause

myself not to believe in evolution by an act of will. I can't look up

at a blue sky and believe it to be green. I account this a strength of

a rationalist.

2) Evolution is still correct regardless of what the two of us do about

our probability assignments. I would need to update my belief because

of a cause that I believe to be uncorrelated with the state of the

actual world.

3) Just before I make my Bayesianitarian act of sacrifice, I will know

even as I do so, correctly and rationally, that evolution is true. And

afterward I'll still know, deep down, whatever my lips say...

4) I have other beliefs about biology that would be inconsistent with a

probability assignment to evolution of 0.5.

5) I do wish to be an altruist, but in the service of that wish, it is

rather more important that I get my beliefs right about evolution than

that J. Random Creationist do so, because JRC is less likely to like

blow up the world an' stuff if he gets his cognitive science wrong.

If I were an utterly selfish Bayesianitarian, how would I maximize only

my own Bayesian score? It is this question that is the foundation of

*probability* theory, the what-we-deep-down-know-will-happen-next,

whatever decisions we make in the service of altruistic utilities.

Point (2) from my previous post should now also be clearer:

* >> 2) Just because, if everyone agreed to do X without further argument
*

* >> or modification (where X is not agreeing to disagree), the average
*

* >> Bayesian score would increase relative to its current position, it
*

* >> does not follow that X is the *optimal* strategy.
*

Two Aumann agents, to whom other agents' probability estimates are

highly informative with respect to the state of the actual world, could

also theoretically follow the average-together-probability-estimates

algorithm and thereby, yes, improve their summed Bayesian scores from

the former status quo - but they would do worse that way than by

following the actual Aumann rules, which do not lend credence as such to

the other agent's beliefs, but simply treat those beliefs as another

kind of Bayesian signal about the state of the actual world.

Thus, if you want to claim a mathematical result about an expected

individual benefit (let alone optimality!) for rationalists deliberately

*trying* to agree with each other, I think you need to specify what

algorithm they should follow to agreement - Aumann agent?

Bayesianitarian altruist? In the absence of any specification of how

rationalists try to agree with each other, I don't see how you could

prove this would be an expected individual improvement. Setting both

your probability estimates to zero is an algorithm for trying to agree,

but it will not improve your Bayesian score.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Mitchell Howe: "Marc Geddes Request"**Previous message:**Jeff Medina: "Reminder: SIAI Volunteers Meeting Tomorrow @ 3pm EST"**Maybe in reply to:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Next in thread:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:50 MDT
*