From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Sun Oct 24 2004 - 07:44:56 MDT
Maru wrote:
> If what I ask has been beaten to death already, then please
> someone direct me to the appropriate archive threads.
> But Ben, what I asked was basically what part of FAI is least
> supported by evidence, and taken most on faith? (I didn't want to
> say it so baldly because it sounds insulting.)
The weakness is that I am not finished, and the part that must be "taken on
faith" (leaving aside certain objections to your terminology) is that I
shall ever finish. An example: In my lap right now is a spiral paper
notebook whose showing page contains an incomprehensible doodle of lines,
filled and empty boxes, and the words "Fred", "CV", and "Motie". This is
the debris of me trying to figure out how to describe Fred's preference
that a Fred-similar system (say, a collective volition) make a decision,
rather than a Motie (an alien with a nonhuman goal system). I know how to
describe this problem if Fred can exactly simulate CV and Motie to
extrapolate their decisions, and then extrapolate the consequences and
expected utility of CV and Motie's decisions; Fred can then simply choose
directly, by proxy, as predictable as pressing a button. But if Fred
doesn't know the choices faced by CV and Motie, or Fred can't personally
extrapolate the consequences of the choices, or if Fred has only an
abstract model of CV and Motie, then it will take some new theory I haven't
seen to give a step-by-step walkthrough of exactly how Fred computes his
preference for CV-decision over Motie-decision. It's a problem directly
related to the difficulty I posted with Schmidhuber's Godel Machine. I
don't presently know the solution, and it must be "taken on faith", as it
were, that I ever shall know. I may know by the end of this day, or by the
end of this year, or one of the other six billion humans out there may have
already solved the problem and I haven't read the paper, or perhaps this
will be the challenge that ultimately defeats me and I shall never know at
all, though I rather doubt it.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:49 MDT