**From:** Eliezer S. Yudkowsky (*sentience@pobox.com*)

**Date:** Sat Mar 12 2005 - 22:57:32 MST

**Next message:**Kaj Sotala: "Re: Eliezer: unconvinced by your objection to safe boxing of "Minerva AI""**Previous message:**Ben Goertzel: "Lojban and AI"**Maybe in reply to:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Next in thread:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Robin Hanson wrote:

*> Eliezer, you are just writing far too much for me to comment on all of
*

*> it.
*

Yes. I know. You don't have to comment on all of it. I just thought I

should say all of it before you wrote your book, rather than afterward.

I don't think that this issue is simple - you did say you wanted to

write a book on it - so I don't think that the volume of discussion is

inappropriate to the question. I understand that your time is

constrained, as is mine.

If you allege that I don't seem interested in the math, you have to

expect a certain probability of a long answer.

*> If you give me an indication of what your key points are, I will
*

*> try to respond to those points.
*

If I had to select out two points as most important, they would be:

1) Just because perfect Bayesians, or even certain formally imperfect

Bayesians that are still not like humans, *will* always agree; it does

not follow that a human rationalist can obtain a higher Bayesian score

(truth value), or the maximal humanly feasible score, by deliberately

*trying* to agree more with other humans, even other human rationalists.

2) Just because, if everyone agreed to do X without further argument or

modification (where X is not agreeing to disagree), the average Bayesian

score would increase relative to its current position, it does not

follow that X is the *optimal* strategy.

*> For now, I will just make a few
*

*> comments on specific claims.
*

*>
*

*> At 06:40 PM 3/9/2005, Eliezer S. Yudkowsky wrote:
*

*>
*

*>> The modesty argument uses Aumann's Agreement Theorem and AAT's
*

*>> extensions as plugins, but the modesty argument itself is not formal
*

*>> from start to finish. I know of no *formal* extension of Aumann's
*

*>> Agreement Theorem such that its premises are plausibly applicable to
*

*>> humans.
*

*>
*

*> Then see: <http://hanson.gmu.edu/disagree.pdf>For Bayesian Wannabes,
*

*> Are Disagreements Not About Information?
*

*> <http://www.kluweronline.com/issn/0040-5833/>Theory and Decision
*

*> 54(2):105-123, March 2003.
*

(I immediately notice that your proof of Lemma 1 describes a Bayesian

Wannabe as wishing to minimize her expected squared error. Orthodox

statisticians minimize their expected squared error because, like,

that's what orthodox statisticians do all day. As described in TechExp,

Bayesians maximize their expectation of the logarithm of the probability

assigned to the actual outcome, which equates to minimizing expected

squared error when the error is believed to possess a Gaussian

distribution and the prior probability density is uniform.

I don't think this is really important to the general thrust of your

paper, but it deserves noting.)

On to the main issue. These Bayesian Wannabes are still unrealistically

skilled rationalists; no human is a Bayesian Wannabe as so defined. BWs

do not self-deceive. They approximate their estimates of deterministic

computations via guesses whose error they treat as random variables.

I remark on the wisdom of Jaynes who points out that 'randomness' exists

in the map rather than the territory; random variables are variables of

which we are ignorant. I remark on the wisdom of Pearl, who points out

that when our map sums up many tiny details we can't afford to compute,

it is advantageous to retain the Markov property, and hence humans

regard any map without the Markov property as unsatisfactory; we say it

possesses unexplained correlations and hence is incomplete.

If the errors in BWs computations are uncorrelated random errors, the

BWs are, in effect, simple measuring instruments, and they can treat

each other as such, combining their two measurements to obtain a third,

more reliable measurement.

If we assume the computation errors follow a bell curve, we obtain a

constructive procedure for combining the computations of any number of

agents; the best group guess is the arithmetic mean of the individual

guesses.

How long is the Emperor of China's nose?

*>> you say: "If people mostly disagree because they systematically
*

*>> violate the rationality standards that they profess, and hold up for
*

*>> others, then we will say that their disagreements are dishonest." (I
*

*>> would disagree with your terminology; they might be dishonest *or*
*

*>> they might be self-deceived. ...
*

*>
*

*> I was taking self-deception to be a kind of dishonesty.
*

Life would be so much simpler if it were. Being honest is difficult and

often socially unrewarding, but halting self-deception is harder.

*>> ... if Aumann's Agreement Theorem is wrong (goes wrong reliably in the
*

*>> long run, not just failing 1 time out of 100 when the consensus belief
*

*>> is 99% probability) then we can readily compare the premises of AAT
*

*>> against the dynamics of the agents, their updating, their prior
*

*>> knowledge, etc., and track down the mistaken assumption that caused
*

*>> AAT (or the extension of AAT) to fail to match physical reality. ...
*

*>
*

*> This actually seems to me rather hard, as it is hard to observe people's
*

*> priors.
*

Is it hard to observe the qualitative fact of whether or not humans'

priors agree? Well, yes, I suppose, as humans, not being Bayesians,

possess no distinguished priors.

But it is a relatively straightforward matter to tell whether humans

behave like Aumann agents. They don't. Similarly, I think it would be

a relatively straightforward matter to sample whether Aumann agents

indeed all had the same priors, as they believed, if they had agreed to

disagree and therefore the premises stood in doubt. Since Aumann agents

know their priors and can presumably report them.

*>> ... You attribute the great number of extensions of AAT to the
*

*>> following underlying reason: "His [Aumann's] results are robust
*

*>> because they are based on the simple idea that when seeking to
*

*>> estimate the truth, you should realize you might be wrong; others may
*

*>> well know things that you do not."
*

*>> I disagree; this is *not* what Aumann's results are based on.
*

*>> Aumann's results are based on the underlying idea that if other
*

*>> entities behave in a way understandable to you, then their observable
*

*>> behaviors are relevant Bayesian evidence to you. This includes the
*

*>> behavior of assigning probabilities according to understandable
*

*>> Bayesian cognition.
*

*>
*

*> The paper I cite above is not based on having a specific model of the
*

*> other's behavior.
*

The paper you cite above does not yield a constructive method of

agreement without additional assumptions. But then the paper does not

prove agreement *given* a set of assumptions. As far as I can tell, the

paper says that Bayesian Wannabes who agree to disagree about

state-independent computations and who treat their computation error as

a state-independent "random" variable - presumably meaning, a variable

of whose exact value they are to some degree ignorant - must agree to

disagree about a state-independent random variable.

*>> So A and B are *not* compromising between their previous positions;
*

*>> their consensus probability assignment is *not* a linear weighting of
*

*>> their previous assignments.
*

*>
*

*> Yes, of course, who ever said it was?
*

If two people who find that they disagree immediately act to eliminate

their disagreement (which should be "much easier"), what should they

compromise on, if not a weighted mix of their probability distributions

weighted by an agreed-upon estimate of relative rationality on that problem?

*>> ... If this were AAT, rather than a human conversation, then as Fred
*

*>> and I exchanged probability assignments our actual knowledge of the
*

*>> moon would steadily increase; our models would concentrate into an
*

*>> ever-smaller set of possible worlds. So in this sense the dynamics of
*

*>> the modesty argument are most unlike the dynamics of Aumann's
*

*>> Agreement Theorem, from which the modesty argument seeks to derive its
*

*>> force. AAT drives down entropy (sorta); the modesty argument
*

*>> doesn't. This is a BIG difference.
*

*>
*

*> AAT is *not* about dynamics at all. It might require a certain dynamics
*

*> to reach the state where AAT applies, but this paper of mine applies at
*

*> any point during any conversation:
*

*>
*

*> <http://hanson.gmu.edu/unpredict.pdf>Disagreement Is Unpredictable.
*

*> <http://www.sciencedirect.com/science/journal/01651765>Economics Letters
*

*> 77(3):365-369, November 2002.
*

I agree that rational agents will not be able to predict the direction

of the other agent's disagreement. But I don't see what that has to do

with my observation, that human beings who attempt to immediately agree

with each other will not necessarily know more after compromising than

they started out knowing.

*>> The AATs I know are constructive; they don't just prove that agents
*

*>> will agree as they acquire common knowledge, they describe *exactly
*

*>> how* agents arrive at agreement.
*

*>
*

*> Again, see my Theory and Decision paper cited above.
*

As far as I can see, this paper is not constructive, but that is because

it does not start from some set of premises and prove agent agreement.

Rather the paper proves that if Bayesian Wannabes treat their

computation errors as state-independent random variables, then if they

agree to disagree about computations, they must agree to disagree about

state-independent random variables. So in that sense, the paper proves

a non-constructive result that is unlike the usual class of Aumann

Agreement theorems. Unless I'm missing something?

*>>> ... people uphold rationality standards that prefer logical
*

*>>> consistency...
*

*>>
*

*>> Is the Way to have beliefs that are consistent among themselves? This
*

*>> is not the Way, though it is often mistaken for the Way by logicians
*

*>> and philosophers. ...
*

*>
*

*> Preferring consistency, all else equal, is not the same as requiring
*

*> it. Surely you also prefer it all else equal.
*

No! No, I do not prefer consistency, all else equal. I prefer *only*

that my map match the territory. If I have two maps that are unrelated

to the territory, I care not whether they are consistent. Within the

Way, fit to the territory is the *only* thing that I am permitted to

consider.

Michael Wilson remarked to me that general relativity and quantum

mechanics are widely believed to be inconsistent in their present forms,

yet they both yield excellent predictions of physical phenomena. This

is a challenge to find a unified theory because underlying reality is

consistent and therefore there is presumably some *specific* consistent

unified theory that would yield better predictions. It is *not* a

problem because I prefer 'all else being equal' that my map be consistent.

*>> ... agree that when two humans disagree and have common knowledge of
*

*>> each other's opinion ... *at least one* human must be doing something
*

*>> wrong. ...
*

*>> One possible underlying fact of the matter might be that one person is
*

*>> right and the other person is wrong and that is all there ever was to it.
*

*>
*

*> This is *not* all there is too it. There is also the crucial question
*

*> of what exactly one of them did wrong.
*

Okay.

*>> Trying to estimate your own rationality or meta-rationality involves
*

*>> severe theoretical problems ... "Beliefs" ... are not ontological
*

*>> parts of our universe, ... if you know the purely abstract fact that
*

*>> the other entity is a Bayesian reasoner (implements a causal process
*

*>> with a certain Bayesian structure),... how do you integrate it? If
*

*>> there's a mathematical solution it ought to be constructive. Second,
*

*>> attaching this kind of *abstract* confidence to the output of a
*

*>> cognitive system runs into formal problems.
*

*>
*

*> I think you exaggerate the difficulties. Again see the above papers.
*

I think I need to explain the difficulties at greater length. Nevermind.

*>> It seems to me that you have sometimes argued that I should
*

*>> foreshorten my chain of reasoning, saying, "But why argue and defend
*

*>> yourself, and give yourself a chance to deceive yourself? Why not
*

*>> just accept the modesty argument? Just stop fighting, dammit!" ...
*

*>
*

*> I would not put my advice that way. I'd say that whatever your
*

*> reasoning, you should realize that if you disagree, that has certain
*

*> general implications you should note.
*

Perhaps we disagree about what those general implications are?

*>> It happens every time a scientific illiterate argues with a scientific
*

*>> literate about natural selection. ... How does the scientific
*

*>> literate guess that he is in the right, when he ... is also aware of
*

*>> studies of human ... biases toward self-overestimation of relative
*

*>> competence? ... I try to estimate my rationality in detail, instead of
*

*>> using unchanged my mean estimate for the rationality of an average
*

*>> human. And maybe an average person who tries to do that will fail
*

*>> pathetically. Doesn't mean *I'll* fail, cuz, let's face it, I'm a
*

*>> better-than-average rationalist. ... If you, Robin Hanson, go about
*

*>> saying that you have no way of knowing that you know more about
*

*>> rationality than a typical undergraduate philosophy student because
*

*>> you *might* be deceiving yourself, then you have argued yourself into
*

*>> believing the patently ridiculous, making your estimate correct
*

*>
*

*> You claim to look in detail, but in this conversation on this the key
*

*> point you continue to be content to just cite the existence of a few
*

*> extreme examples, though you write volumes on various digressions. This
*

*> is what I meant when I said that you don't seem very interested in
*

*> formal analysis.
*

I don't regard this as the key point. If you regard it as the key

point, then this is my reply: while there are risks in not

foreshortening the chain of logic, I think that foreshortening the

reasoning places an upper bound on predictive power and that there exist

alternate strategies which exceed the upper bound, even after the human

biases are taken into account. To sum up my reply, I think I can

generate an estimate of my rationality that is predictively better than

the estimate I would get by substituting unchanged my judgment of the

average human rationality on the present planet Earth, even taking into

account the known biases that have been discovered to affect

self-estimates of rationality. And this explains my persistent

disagreement with that majority of the population which believes in God

- how do you justify this disagreement for yourself?

The formal math I can find does not deal at all with questions of

self-deceptive reasoning or the choice of when to foreshorten a chain of

reasoning with error-prone links. Which is the formal analysis that you

feel I am ignoring?

*> Maybe there are some extreme situations where it is "obvious" that one
*

*> side is right and the other is a fool.
*

How do these extreme situations fit into what you seem to feel is a

mathematical result requiring agreement? The more so, as, measuring

over Earth's present population, most cases of "obviousness" will be

wrong. Most people think God obviously exists.

*> This possibility does not
*

*> justify your just disagreeing as you always have.
*

I started disagreeing differently after learning that Bayesians could

not agree to disagree, though only when arguing with people I regarded

as aspiring rationalists who had indicated explicit knowledge of

Aumann-ish results. Later I would launch a project to break my mind of

the habit of disagreeing with domain experts unless I had a very strong

reason. Perhaps I did not adjust my behavior enough; I do not say,

"See, I adjusted my behavior!" as my excuse. Let the observation just

be noted for whatever the information is worth.

*> The question is what
*

*> reliable clues you have to justify disagreement in your typical
*

*> practice. When you decide that your beliefs are better than theirs,
*

*> what reasoning are you going through at the meta-level? Yes, you have
*

*> specific arguments on the specific topic, but so do they - why exactly
*

*> is your process for producing an estimate more likely to be accurate
*

*> than their process?
*

Sometimes it isn't. Then I try to substitute their judgment for my

judgment. Then there isn't a disagreement any more. Then nobody

remembers this event because it flashed by too quickly compared to the

extended disagreements, and they call me stubborn.

I do reserve to myself the judgment of when to overwrite my own opinion

with someone else's. Maybe if someone who knew and understood Aumann's

result, and knew also to whom they spoke, said to me, "I know and

respect your power, Eliezer, but I judge that in this case you must

overwrite your opinion with my own," I would have to give it serious

consideration.

If you're asking after specifics, then I'd have to start describing the

art of specific cases, and that would be a long answer. The most recent

occasion where I recall attempting to overwrite my own opinion with

someone else's was with an opinion of James Rogers's. That was a case

of domain-specific expertise; James Rogers is a decent rationalist with

explicit knowledge of Bayesianity but he hasn't indicated any knowledge

of Aumannish things. Maybe I'll describe the incident later if I have

the time to write a further reply detailing what I feel to be the

constructive art of resolving disagreements between aspiring rationalists.

Michael Raimondi and I formed a meta-rational pair from 2001 to 2003.

We might still be a meta-rational pair now, but I'm not sure.

*> In the above you put great weight on literacy/education, presuming that
*

*> when two people disagree the much more educated person is more likely to
*

*> be correct.
*

In this day and age, people rarely go about disagreeing as to whether

the Earth goes around the Sun. I would attribute the argument over

evolution to education about a simple and enormously overdetermined

scientific fact, set against tremendous stupidity and self-deception

focused on that particular question.

It's not a general rule about the superiority of education - just one

example scenario. If you want an art of resolving specific

disagreements between rationalists, or cues to help you estimate how

likely you are to be correct on a particular question, then the art is

specific and complicated.

That said, I do indeed assign tremendous weight to education. Degree of

education is domain-specific; an educated biologist is not an educated

physicist. The value of education is domain-specific; not all education

is equally worthwhile. If a physicist argues with a biologist about

physics then the biologist's opinion has no weight. If a clinical

psychologist argues with a physicist about psychology then, as far as

any experimental tests have been able to determine, the clinical

psychologist has no particular advantage.

*> Setting aside the awkward fact of not actually having hard
*

*> data to support this, do you ever disagree with people who have a lot
*

*> more literacy/education than you?
*

Define "literacy". I strive to know the basics of a pretty damn broad

assortment of fields. *You* might (or might not) have greater breadth

than I, but most of your colleagues' publication records haven't nearly

your variety.

*> If so, what indicators are you using
*

*> there, and what evidence is there to support them?
*

When I disagree with an 'educated' person, it may be because I feel the

other person to be ignorant of specific known results; overreaching his

domain competence into an external domain; affected by wishful thinking;

affected by political ideology; educated but not very bright; a

well-meaning but incompetent rationalist; or any number of reasons. Why

are the specific cues important to this argument? You seem to be

arguing that there are mathematical results which a priori rule out the

usefulness of this digression.

*> A formal Bayesian analysis of such an indicator would be to construct a
*

*> likelihood and a prior, find some data, and then do the math. It is not
*

*> enough to just throw out the possibility of various indicators being
*

*> useful.
*

I lack the cognitive resources for a formal Bayesian analysis, but my

best guess is that I can do better with informal analysis than with no

analysis. As the Way renounces consistency for its own sake, so do I

renounce formality, save in the service of arriving to the correct answer.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Kaj Sotala: "Re: Eliezer: unconvinced by your objection to safe boxing of "Minerva AI""**Previous message:**Ben Goertzel: "Lojban and AI"**Maybe in reply to:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Next in thread:**Eliezer S. Yudkowsky: "Re: Overconfidence and meta-rationality"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:50 MDT
*