From: Matt Mahoney (matmahoney@yahoo.com)
Date: Tue May 19 2009 - 08:56:36 MDT
--- On Mon, 5/18/09, Mike Dougherty <msd001@gmail.com> wrote:
> Benja Fallenstein
> <benja.fallenstein@gmail.com>
> wrote:
> > On day 1, give the agent a choice between two contracts, A and B. A
> > pays $1 every day until the end of time. B offers no recurrent
> > payments, but on every successive day the agent can terminate the
> > contract; if the agent terminates the contract on day n, it receives a
> > one-time payment of $(n+1000).
>
> A mortal may wisely determine that there is a high likelihood they
> could outlive their prediction by more than 1000 days, so should
> choose contract A the same as an immortal who chooses A.
I think this is a valid solution, assuming an agent will always prefer infinite utility over any finite utility. To address your concern, change contract B to pay $(2^n) or $(n^^^^n).
> Every scenario I have imagined to 'solve' this puzzle involves a
> cooperative relationship between two conscious agents gaming a dumb
> system. That's so far from the scenario that it isn't worth
> discussing. I still conclude that "immortality" is NOT computable.
> (for the commonly understood definition)
My original assumption is that the agent is able to know or learn the environment and compute the optimal solution. I realize that the general case is not computable. So the answer to the question "is there an agent that behaves rationally consistent with a belief in (im)mortality in all environments?" is no because computing a rational (optimal) solutions is not computable. My question is "are there environments in which mortality-believing and immortality-believing rational agents can be distinguished?" and the answer is yes.
Now here is another question. Can we distinguish an agent that is certain of its immortality from one that doesn't know for sure? Note that an agent that believes it is immortal with any probability p > 0 will choose contract A because its expected utility is infinite and B is finite regardless of how fast the payout grows.
-- Matt Mahoney, matmahoney@yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:04 MDT