From: Wei Dai (weidai@weidai.com)
Date: Mon Sep 08 2008 - 03:31:20 MDT
Norman Noman wrote:
> How is an AI which flips a coin and then turns the universe into
> paperclips
> or staples depending on the result "irrational"? It's certainly odd, but
> it
> seems rational to me.
It's "irrational" in the sense that such an AI can't be modeled by standard
decision theory, which is commonly used as a mathematical definition of
rationality. I tried to explain why in my post, but maybe didn't get my
point across. In standard decision theory, an agent either prefers one state
to another, say A and B (i.e., assigns them equal utilities), in which case
it also prefers state A to any probabilistic mixture of states A and B, or
it is indifferent between A and B, in which case it is also indifferent
between all probabilistic mixtures of states A and B. Note that it cannot
strictly prefer a probabilistic mixture of A and B to both A and B, which is
what is needed to get the "coin flip" behavior from the AI.
I agree with you that this AI should not be considered irrational, which is
why I'm arguing that we need an alternative to standard decision theory.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT