**From:** Eliezer S. Yudkowsky (*sentience@pobox.com*)

**Date:** Tue Sep 16 2003 - 16:09:06 MDT

**Next message:**AD.COM - Marcos Guillen: "RE: [Fwd: [>Htech] Artificial Development to Build World's Biggest Spiking Neural Network]"**Previous message:**Wei Dai: "Re: My doubts about Libertarianism and volitional morality"**In reply to:**Wei Dai: "Re: My doubts about Libertarianism and volitional morality"**Next in thread:**Wei Dai: "Re: Constructing volition (was: my doubts)"**Reply:**Wei Dai: "Re: Constructing volition (was: my doubts)"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Wei Dai wrote:

*> [Not cc'ing wta-talk because I'm not on that list.]
*

*>
*

*> On Thu, Sep 11, 2003 at 04:55:24PM -0400, Eliezer S. Yudkowsky wrote:
*

*>
*

*>>Suppose we assume that X has an objective frequency given A, F(x|a). If
*

*>>the subjective frequency assigned by the person to P(x|a) doesn't match
*

*>>the objective frequency F(x|a), then to a first approximation we can say
*

*>>that the subject's "volition" as a moral desideratum should be computed
*

*>>using U(x)F(x|a), while the subject's actual decisions will in fact be
*

*>>computed using U(x)P(x|a). In other words, your "volition" is an abstract
*

*>>entity which your actual decisions only approximate; your volition is the
*

*>>decision you would make if you had perfect information.
*

*>
*

*> Are you aware that given any utility function U and probability
*

*> function P, it's possible to find an infinite set of different pairs of
*

*> functions U' and P', such that the preference ordering over choices
*

*> determined by U' and P' is the same as the preference ordering
*

*> determined by U and P? Someone with U' and P' would behave identically
*

*> to someone with U and P in every possible situation.
*

*>
*

*> Therefore, there's no objective method to separate a person's
*

*> actual preferences into a value component and a belief component, and
*

*> then to compute what his preferences would be if he had "correct"
*

*> beliefs.
*

*>
*

*> How would volitionism deal with this problem?
*

Classical economists tend to construct U(x) by looking at a person's

actual choices, assuming their probability function P obeys Bayesian

rules, and assuming that their choices obey the expected utility rule.

Cognitive psychologists know darn well that people don't use Bayesian

reasoning on probabilities and that their choices aren't consistent under

expected utility, so they try to figure out the probabilities actually

assign, and the forces that actually influence their choices.

As humans, we simply don't *have* a U(x) in the sense of the expected

utility rule - we are not that consistent. We do, however, have a

decision system that relates utilities to outcomes, outcomes to

probabilities, probabilities to actions, and uses all that information to

assign desirabilities to actions. We just don't do it consistently.

Some analyses of utility assume that humans are black boxes and that we

are allowed only to look at their actions, and that we are supposed to

deduce a utility function and probability assignment from these actions

and the (false) assumption that people are perfectly structured according

to Bayesian probability and expected utility. If so, there would be an

infinite set of U', P' that could generate those choices, but that might

not generate any ambiguity in the set of possible *volitional* orderings

over the same choices. For example, if you multiply all utilities by a

non-negative constant, you get the same preference orderings. You also

get the same volitional orderings. So the apparently infinite range of

U(x) merely reflects the fact that utility is a measure rather than a

function. (That is, the "utility" in "expected utility", which is a

mathematical construct, is a measure rather than a function; this is not

intended as an assertion about human cognition.)

I haven't looked into the result you refer to, Wei Dai, but my initial

impression is that it assumes infinite degrees of freedom in both U(x) and

P(x). Leaving aside the former, the latter, at least, is usually assumed

to normatively obey Bayesian probabilities, and cognitively it obeys

certain loose non-Bayesian rules. We are surprised when we see that under

certain circumstances people assign subjective likelihoods P(A&B) > P(A),

but having discovered this, we can then predict that most people will do

it most of the time under those conditions. So there are not infinite

degrees of freedom in P(x), either normatively or cognitively, and if you

use this constraint to construct U(x) you will not find infinite relevant

degrees of freedom in U(x) either. Of course I am only reasoning

intuitively here, and I may have gotten the math wrong.

But, mostly my reply is that I'm not using the economist assumption that

we have to construct U(x) and P(x) by looking exclusively at people's

choices. From a volitionist standpoint, what I would like to do is open

up people's heads and look at their mind-state, figure out what systems

are working, what they contain, what actual system is producing the

choices, and then, having this functional decomposition, ask which parts

have U(x) nature and which parts have P(x) nature, bearing in mind that

they will overlap. Existing cognitive psychology actually goes quite a

ways toward doing this.

Or to put it another way... suppose I show someone a spinner wheel A that

is one-tenth red, and another spinner wheel B that is nine-tenths red. I

ask: "Would you rather have a chocolate ice cream cone given that A shows

red, or a vanilla ice cream cone given that B shows red?" Technically we

should also show an empty spinner C, in case the person is on a diet and

doesn't want an ice cream cone at all. Suppose the person waves aside our

offer to opt out of the experiment.

If the person then says they would rather have a chocolate ice cream cone

on A, rather than a vanilla ice cream cone on B, it *could* be that they

like vanilla more than chocolate, but that they think that B is extremely

unlikely to come up red for some odd reason, or that they think A is

unusually likely to come up red for some odd reason. But most people,

modeling what is actually likely to be going on inside the chooser's head

- inferentially, rather than deductively - would make the pragmatic guess

that the person prefers chocolate to vanilla.

And if we opened up the person's mind-state and looked inside to see what

kind of information-processing was going on, we would find some subjective

likelihood for A and B was being generated, some expected liking was being

generated for vanilla and chocolate, and that these two cognitive datums

interacted to generate desirabilities for the actions and finally a choice.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**AD.COM - Marcos Guillen: "RE: [Fwd: [>Htech] Artificial Development to Build World's Biggest Spiking Neural Network]"**Previous message:**Wei Dai: "Re: My doubts about Libertarianism and volitional morality"**In reply to:**Wei Dai: "Re: My doubts about Libertarianism and volitional morality"**Next in thread:**Wei Dai: "Re: Constructing volition (was: my doubts)"**Reply:**Wei Dai: "Re: Constructing volition (was: my doubts)"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:42 MDT
*