**From:** Wei Dai (*weidai@weidai.com*)

**Date:** Sun Sep 07 2008 - 16:36:48 MDT

**Next message:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Previous message:**m.l.vere@durham.ac.uk: "[sl4] Please unsubscribe me from these mailings"**Next in thread:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Norman Noman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Maybe reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Tim Freeman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Tim Freeman: "Different priors (was Re: [sl4] Bayesian rationality vs. voluntary mergers)"**Reply:**Eliezer Yudkowsky: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Rolf Nelson: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

After suggesting in a previous post [1] that AIs who want to cooperate with

each other may find it more efficient to merge than to trade, I realized

that voluntary mergers do not necessarily preserve Bayesian rationality,

that is, rationality as defined by standard decision theory. In other words,

two "rational" AIs may find themselves in a situation where they won't

voluntarily merge into a "rational" AI, but can agree merge into an

"irrational" one. This seems to suggest that we shouldn't expect AIs to be

constrained by Bayesian rationality, and that we need an expanded definition

of what rationality is.

Let me give a couple of examples to illustrate my point. First consider an

AI with the only goal of turning the universe into paperclips, and another

one with the goal of turning the universe into staples. Each AI is

programmed to get 1 util if at least 60% of the accessible universe is

converted into its target item, and 0 utils otherwise. Clearly they can't

both reach their goals (assuming their definitions of "accessible universe"

overlap sufficiently), but they are not playing a zero-sum game, since it is

possible for them to both lose, if for example they start a destructive war

that devastates both of them, or if they just each convert 50% of the

universe.

So what should they do? In [1] I suggested that two AIs can create a third

AI whose utility function is a linear combination of the utilities of the

original AIs, and then hand off their assets to the new AI. But that doesn't

work in this case. If they tried this, the new AI will get 1 util if at

least 60% of the universe is converted to paperclips, and 1 util if at least

60% of the universe is converted to staples. In order to maximize its

expected utility, it will pursue the one goal with the highest chance of

success (even if it's just slightly higher than the other goal). But if

these success probabilities were known before the merger, the AI whose goal

has a smaller chance of success would have refused to agree to the merger.

That AI should only agree if the merger allows it to have a close to 50%

probability of success according to its original utility function.

The problem here is that standard decision theory does not allow a

probabilistic mixture of outcomes to have a higher utility than the

mixture's expected utility, so a 50/50 chance of reaching either of two

goals A and B cannot have a higher utility than 100% chance of reaching A

and a higher utility than 100% chance of reaching B, but that is what is

needed in this case in order for both AIs to agree to the merger.

The second example shows how a difference in the priors of two AIs, as

opposed to their utility functions, can have a similar effect. Suppose two

AIs come upon an alien artifact which looks like a safe with a combination

lock. There is a plaque that says they can try to open the lock the next

day, but it will cost $1 to try each combination. Each AI values the

contents of the safe at 3 utils, and the best alternative use of the $1 at 2

utils. They also each think they have a good guess of the lock combination,

assigning a 90% probability of being correct, but their guesses are

different due to having different priors. They have until tomorrow to decide

whether to try their guesses or not, but in the mean time they have to

decide whether or not to merge. If they don't merge, they will each try a

guess and expect to get .9*3=2.7 utils, but if they do merge into a new

Bayesian AI with an average of their priors, the new AI will assign .45

probability of each guess being correct, and since the expected utility of

trying a guess is now .45 * 3 < 2, it will decide not to try either

combination. The original AIs, knowing this, would refuse to merge.

To generalize a bit from these examples, it appears that standard decision

theory was created to model or guide the decision making processes of

individuals, whereas AIs may need to represent the beliefs and interests of

groups. This may be the result of mergers as in these examples, or the AIs

may be designed from the start to benefit whole groups or societies. The

standard Bayesian notion of rationality does not seem adequate for this

task.

[1] "trade or merge?"

http://www.nabble.com/-sl4--trade-or-merge--td18500141.html

**Next message:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Previous message:**m.l.vere@durham.ac.uk: "[sl4] Please unsubscribe me from these mailings"**Next in thread:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Byrne Hobart: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Norman Noman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Maybe reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Tim Freeman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Tim Freeman: "Different priors (was Re: [sl4] Bayesian rationality vs. voluntary mergers)"**Reply:**Eliezer Yudkowsky: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Reply:**Rolf Nelson: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*