From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Nov 24 2007 - 15:08:42 MST
Matt Mahoney wrote:
> I agree it will wipe out the human race, but I disagree that compression is
> not CEV. CEV is a definition of friendliness, not a solution.
>
> Using http://www.overcomingbias.com/2007/11/complex-wishes.html as an example,
> a good data compressor will know what you mean when you say "get my mother out
> of the burning building", in the sense that if
>
> s1 = "'Do you mean to get her out alive?' 'Yes'"
> s2 = "'Do you mean to get her out alive?' 'No'"
>
> then string s1 has a higher probability, and therefore a shorter code length
> than s2, and likewise for related questions with common sense answers, because
> common sense knowledge improves compression of dialog with common sense
> answers.
If you define the extrapolation of a 'Yes' answer in dialogue as your
utility function, that creates instrumental pressure to emit extremely
persuasive arguments or even to simply reprogram brains to answer
'Yes', there being no hard line between these two options.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:01 MDT