From: Ben Goertzel (ben@goertzel.org)
Date: Wed Feb 22 2006 - 06:10:36 MST
On 2/22/06, Jeff Medina <analyticphilosophy@gmail.com> wrote:
> William Pearson wrote:
> > So the maximum number of bits of Phi and hence number of known or
> > provably friendly programs is bounded by the length of your starting
> > program.
>
> As Eliezer mentioned, the number is infinite, and hence unbounded
It seems more interesting to me to observe that the maximum
algorithmic information of any program in the set of provably friendly
programs is finite, and is bounded by the length of your starting
program. (here "provably" is taken to mean "provably by your starting
program")
(The algorithmic information of X is defined, loosely, as the length
of the shortest program for computing X)
On the other hand, if the starting program is allowed to increase its
algorithmic information over time via adding new physical compute
resources, then things get more complicated -- but this doesn't get
around the basic problem and obvious I've cited above.
Chaitin has summarized Godel's theorem nicely as "You can't use a ten
pound formal system to prove a twenty pound theorem."
The application of this to Friendly AI would seem to be: "A ten pound
intelligence can't prove the Friendliness of a twenty pound
intelligence."
Furthermore, it seems to me that "A ten pound intelligence can't prove
any solid probabilistic bounds on the Friendliness of a twenty pound
intelligence" either....
This suggests that *perhaps* mathematical proof is not the right
paradigm for us to be using to think about these issues... (Remember,
"mathematical proof" is just an abstract construct found useful by us
humans in particular situations we have encountered so far...)
-- Ben
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT