**From:** Eliezer Yudkowsky (*sentience@pobox.com*)

**Date:** Sun Dec 07 2008 - 16:12:52 MST

**Next message:**Petter Wingren-Rasmussen: "Re: [sl4] How to imprint rules (re:AI's behaving badly)"**Previous message:**Peter de Blanc: "Re: [sl4] Convergence of Expected Utilities with Algorithmic Probability Distributions - uh?"**In reply to:**Peter de Blanc: "Re: [sl4] Convergence of Expected Utilities with Algorithmic Probability Distributions - uh?"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

On Sun, Dec 7, 2008 at 11:50 AM, Peter de Blanc <peter@spaceandgames.com> wrote:

*>
*

*> So to consider a slightly simpler case than in the paper, suppose p and U
*

*> were both computable functions. p is a function of an _index_ of a program,
*

*> whereas U is a function of the _output_ of the program. Since the map from
*

*> programs to outputs is not a total computable function, it should seem
*

*> conceivable that U(program_n(0)) could grow more quickly than 1/p(n),
*

*> because the former is not a total function in n but the latter is.
*

Ah, I had a similar question when reading the paper. I haven't gone

through it in depth, but need to do so at some point. From this I

suspect that my reply may be some variant of, "If you know the output,

you can take the output into account as information in assessing the

probability that you exert unique control over that universe rather

than being within that universe; if you don't know the output, your

utility function can't be run over it either."

-- Eliezer Yudkowsky Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Petter Wingren-Rasmussen: "Re: [sl4] How to imprint rules (re:AI's behaving badly)"**Previous message:**Peter de Blanc: "Re: [sl4] Convergence of Expected Utilities with Algorithmic Probability Distributions - uh?"**In reply to:**Peter de Blanc: "Re: [sl4] Convergence of Expected Utilities with Algorithmic Probability Distributions - uh?"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*