From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Thu Jun 17 2004 - 19:42:12 MDT
Mike wrote:
> On the WIKI, you said "To construe your volition, I need to define a
> dynamic for extrapolating your volition, given knowledge about you." So
> I asked the question several times of how you were going to collect that
> knowledge. Now you're saying all you need is a definition of future
> collective volition, and the AI will make it all right? I don't see
> where MY current volition ever enters into that equation.
If a Bayesian Thingy can correctly, confidently guess the entire
extrapolation by watching downloaded anime, it means that humans are
sufficiently alike under the skin that your extrapolated volition can be
guessed simply by knowing that you are human - or that 95% of humanity fits
this description, and the direction of our social growth predictably
doesn't hinge on the remaining 5%. I do not call this probable, nor
improbable, but for myself I would like to know that the collective
volition was actually extrapolating my individual details, even if it
involved only a small information gain, because it would lend a sense of
participation.
Every time someone guesses What The Collective Volition Will Do Afterward,
they are in essence making a prediction of output given knowledge only of
human nature, not of six billion specific humans. I feel confident in
saying, without doing the interviews, that we are not likely to wish the
solar system transformed into paperclips.
Is it possible that you misunderstood what I meant, and heard that the
programmers would be defining the output of the collective volition, rather
than the extrapolation process of the initial dynamic? The programmers are
saying things like what it means for a human to "want" something and how to
locate "humans" in external reality - not hardwiring "humans want food",
albeit it is the sort of thing one might use as a test case. Test cases
aren't part of the initial dynamic's defined invariant, where "want" and
"human" are.
But, given a well-defined extrapolation dynamic, reading evolutionary
psychology papers and watching downloaded anime is sufficient information
to guess that most humans like eating food - you don't need to interview
every single one of six billion humans and ask, although you can interview
if your decisions are influenced by the exact degree of total liking, or if
the individual humans would want to be asked because it lends a sense of
participation, or if you don't trust the downloaded anime and ev-psych
papers. If collective volition survives as the successor dynamic, then you
might want to extrapolate every individual in detail just because that's
their inalienable political right.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT