**From:** Marc Geddes (*marc_geddes@yahoo.co.nz*)

**Date:** Fri Jan 21 2005 - 23:12:12 MST

**Next message:**Thomas Buckner: "Re: maybe we are in pi"**Previous message:**Marc Geddes: "Re: My attempt at a general technical definition of 'Friendliness'"**In reply to:**Eliezer Yudkowsky: "Technical definitions of FAI"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

--- Eliezer Yudkowsky <sentience@pobox.com> wrote:

*>
*

*> This breaks down FAI into four problems:
*

*>
*

*> 1) Devise a technical specification of a class of
*

*> invariants, such that it
*

*> is possible to prove a recursively self-improving
*

*> optimizer stays within
*

*> that invariant.
*

*> 2) Given an invariant, devise an RSIO such that it
*

*> proves itself to follow
*

*> the invariant. (The RSIO's proof may be, e.g., a
*

*> proof that the RSIO is
*

*> stable if Peano Arithmetic is consistent.)
*

is/does. being/becoming. What perhaps you have

failed to realize is that the *process* of proof may

in the end be no different to the proof itself.

Is there, ultimately, any distinction between the

mathematical, the physical and the perceptual worlds?

I think the proof that the RSIO follows the invariant

is uncomputable. But it can still be approximated to

any desired probability of success. And what do I

think the 'proving process' is? In the end I don't

think it's any different to the 'invariant' itself.

*> 3) Devise a framework and a formal verification
*

*> protocol for translating a
*

*> human intention (e.g. "implement the collective
*

*> volition of humankind")
*

*> into an invariant. This requirement interacts
*

*> strongly with (1) because
*

*> the permitted class of invariants has to be able to
*

*> represent the output of
*

*> the protocol.
*

*> 4) Intend something good.
*

Here is my (non-technical) specification of what I

think the invariant is/what it does/is good. is/does.

being/becoming. Again, are you sure that there is

any real distinction between these terms? Are you

sure that (1), (2) (3) and (4) are really seperate

from each other? Think about it!

The Good thing I intend:

http://www.sl4.org/wiki/TheWay

=====

Find local movie times and trailers on Yahoo! Movies.

http://au.movies.yahoo.com

**Next message:**Thomas Buckner: "Re: maybe we are in pi"**Previous message:**Marc Geddes: "Re: My attempt at a general technical definition of 'Friendliness'"**In reply to:**Eliezer Yudkowsky: "Technical definitions of FAI"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:50 MDT
*