From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue Feb 21 2006 - 19:42:55 MST
William Pearson wrote:
>
> So the maximum number of bits of Phi and hence number of known or
> provably friendly programs is bounded by the length of your starting
> program. It would be less than that because you would have to take
> into consideration the bit requirement of the machinery to transform
> your bit of Phi into a program and other things you want your program
> to do. But tightening the bound is not of much interest I don't think.
>
> Anything wrong with this semi-formal proof?
Looks good to me. I only have two caveats:
1) The space is circumscribed by the initial bits but still infinite.
There is an obvious infinite class of Friendly programs which are all
the same except for containing N bits of unreachable code.
2) If the 'Friendly' invariant includes dependencies on outside
physical processes which can affect the approval of new programs, e.g.,
coherent extrapolated volition contains a dependency on the cognitive
states of humanity, then the bound does not formally hold because new
bits are absorbed from the environment.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT