From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Wed Jun 02 2004 - 03:32:38 MDT
Wei Dai wrote:
> What does "knew more" mean? For example, the majority of human beings in
> existence today do not "know" that humans are the product of biological
> evolution. As another example, they believe that their own religious
> beliefs are correct and that the religious beliefs of most other people
> are incorrect. In trying to extrapolate their true volitions, would you
> replace their belief in creationism with belief in evolution? Would you
> replace their belief in a particular God with a Bayesian probability
> distribution over a wide range of possible gods? If the answer is "yes",
> how would you then convince them that the extrapolated volitions is in any
> sense *theirs*?
The last question strikes me as irrelevant; insofar as I understand these
matters, it is not only unnecessary to convince people that their volition
represents their personal opinion, it is actively harmful. I'd guess that
no one should ever be told what their volition is. If they argue, they are
driven away from a good answer. If they accept, you have an infinite
recursion problem, where the volition asks what you'll think when you grow
up, and the volition can return any answer, and you'll accept it because
hey, it's your volition. Like a time travel paradox where the inventor of
the time machine goes back and tells herself how to do it. It looks
straightforward enough to choose an order of evaluation that prevents this
in the volition, but you see the human problem? Also, your volition will
probably spread, and the answer given one that satisfices the spread,
possibly not a *best* answer for any real individual. And collective
volition runs on many superposed spreads, not any individual volition.
If this causes young humans to indignantly reject the collective volition -
what, it's claiming to act on our behalf, and we can't even be told what
our volition is? - that might also be a good idea. A devout atheist like
myself would hate to have created someone's God, and it probably wouldn't
be good for humanity.
But that is all speculation, so let us return to your right and proper
question about dynamics.
Let me toss your question back to you: What do you think a devout
Christian should be said to *want*, conditional upon Christianity being
false? Fred wants box A conditional upon box A containing the dimaond;
Fred wants box B conditional upon box B containing the diamond. What may a
devout Christian be said to want, conditional upon Christianity being
false? I can think of several approaches. The human approach would be to
*tell* the devout Christian that Christianity was false, then accept what
they said in reply; but that is the Christian's reaction on being *told*
that Christianity is false, it is not what the Christian "would want"
conditional upon Christianity being false. If the Christian is capable of
seriously thinking about the possibility, the problem is straightforward
enough. If not, how would one extract an answer for the conditional question?
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:38 MST