Re: ESSAY: Program length, Omega and Friendliness

From: Eliezer S. Yudkowsky (
Date: Tue Feb 21 2006 - 19:42:55 MST

William Pearson wrote:
> So the maximum number of bits of Phi and hence number of known or
> provably friendly programs is bounded by the length of your starting
> program. It would be less than that because you would have to take
> into consideration the bit requirement of the machinery to transform
> your bit of Phi into a program and other things you want your program
> to do. But tightening the bound is not of much interest I don't think.
> Anything wrong with this semi-formal proof?

Looks good to me. I only have two caveats:

1) The space is circumscribed by the initial bits but still infinite.
There is an obvious infinite class of Friendly programs which are all
the same except for containing N bits of unreachable code.

2) If the 'Friendly' invariant includes dependencies on outside
physical processes which can affect the approval of new programs, e.g.,
coherent extrapolated volition contains a dependency on the cognitive
states of humanity, then the bound does not formally hold because new
bits are absorbed from the environment.

Eliezer S. Yudkowsky                
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT