**From:** Eliezer Yudkowsky (*sentience@pobox.com*)

**Date:** Thu Oct 28 2004 - 18:36:20 MDT

**Next message:**Ben Goertzel: "RE: Universal ethics"**Previous message:**David Clark: "Re: Universal ethics"**In reply to:**Slawomir Paliwoda: "Re: Donate Today and Tomorrow"**Next in thread:**Marc Geddes: "Re: SIAI: Donate Today and Tomorrow"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Slawomir Paliwoda wrote:

*>
*

*> There's no question that SIAI's work on FAI *theory* is important and
*

*> should go on, but unless Seed AI team translates Friendliness into a
*

*> mathematical model and proves that it would be impossible for the entity
*

*> to ever turn unFriendly, we should be extremely nervous about any plans
*

*> to *implement* that theory. The stakes are simply too high.
*

I was planning to put a mathematical upper bound on the probability of

catastrophic failure within an expected operational lifetime before

replacement. Impossibility is not a word to be used outside the context of

physics.

Of course this just passes the buck of the fundamental safety problem,

which then becomes defining "catastrophic failure" to include all

catastrophic failures. Crying mathematical certainty doesn't help with

this; it just excludes silly solutions that the speaker can't translate

into silly math. If you make a big deal out of whether it's mathematical

or not, you'll just get silly solutions translated into silly math and the

speaker will say: lo, it is math. Rather go on attacking the fundamental

safety problem in its new form, broadening your understanding of failure,

seeing how simple definitions fail in simple loopholes.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Ben Goertzel: "RE: Universal ethics"**Previous message:**David Clark: "Re: Universal ethics"**In reply to:**Slawomir Paliwoda: "Re: Donate Today and Tomorrow"**Next in thread:**Marc Geddes: "Re: SIAI: Donate Today and Tomorrow"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:49 MDT
*