**From:** Wei Dai (*weidai@weidai.com*)

**Date:** Mon Sep 08 2008 - 03:31:20 MDT

**Next message:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Previous message:**Norman Noman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**In reply to:**Wei Dai: "[sl4] Bayesian rationality vs. voluntary mergers"**Next in thread:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Norman Noman wrote:

*> How is an AI which flips a coin and then turns the universe into
*

*> paperclips
*

*> or staples depending on the result "irrational"? It's certainly odd, but
*

*> it
*

*> seems rational to me.
*

It's "irrational" in the sense that such an AI can't be modeled by standard

decision theory, which is commonly used as a mathematical definition of

rationality. I tried to explain why in my post, but maybe didn't get my

point across. In standard decision theory, an agent either prefers one state

to another, say A and B (i.e., assigns them equal utilities), in which case

it also prefers state A to any probabilistic mixture of states A and B, or

it is indifferent between A and B, in which case it is also indifferent

between all probabilistic mixtures of states A and B. Note that it cannot

strictly prefer a probabilistic mixture of A and B to both A and B, which is

what is needed to get the "coin flip" behavior from the AI.

I agree with you that this AI should not be considered irrational, which is

why I'm arguing that we need an alternative to standard decision theory.

**Next message:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Previous message:**Norman Noman: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**In reply to:**Wei Dai: "[sl4] Bayesian rationality vs. voluntary mergers"**Next in thread:**Wei Dai: "Re: [sl4] Bayesian rationality vs. voluntary mergers"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*