**From:** Eliezer S. Yudkowsky (*sentience@pobox.com*)

**Date:** Tue May 09 2006 - 17:29:51 MDT

**Next message:**H C: "Interesting Conversation"**Previous message:**Mikko Särelä: "Re: Changing the value system of FAI"**In reply to:**Mikko Särelä: "Re: Changing the value system of FAI"**Next in thread:**H C: "Interesting Conversation"**Reply:**H C: "Interesting Conversation"**Reply:**Dani Eder: "Re: Changing the value system of FAI"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Mikko Särelä wrote:

*>
*

*> My guess is that he is stating that you cannot prove any possible self
*

*> modification to either to be conclusively good or bad. Where good or bad
*

*> is judged by any reasonable criterion. Which means that you either have to
*

*> narrow your self modifications to those that can be proven good, or to
*

*> make some sort of guesses and jump to a possible unknown.
*

That you can't prove this for every possible self-modification is

obvious by Rice's Theorem, an extension of the halting problem which

says that if a property of a program's output is nontrivial, it is not

possible to say in finite time whether any possible program has that

output. This includes things like simple multiplication.

Nonetheless human engineers build CPUs that carry out multiplication, by

working in the restricted space of CPU designs that the chip engineer

*knows* carries out multiplication, which is a subset of all possible

chip designs that carry out multiplication. To build a bridge that

stays up, it is not necessary to be able to calculate of any possible

bridge whether it stays up. It is enough to know at least one bridge

design such that you can calculate that it supports at least 30 tons.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**H C: "Interesting Conversation"**Previous message:**Mikko Särelä: "Re: Changing the value system of FAI"**In reply to:**Mikko Särelä: "Re: Changing the value system of FAI"**Next in thread:**H C: "Interesting Conversation"**Reply:**H C: "Interesting Conversation"**Reply:**Dani Eder: "Re: Changing the value system of FAI"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:56 MDT
*