From: Matt Mahoney (matmahoney@yahoo.com)
Date: Sat Nov 24 2007 - 13:12:52 MST
Tom McCabe wrote:
> This is not a recipe for CEV, or for anything besides getting us all
> killed in nasty ways. See http://www.acceleratingfuture.com/tom/?p=71.
I agree it will wipe out the human race, but I disagree that compression is
not CEV. CEV is a definition of friendliness, not a solution.
Using http://www.overcomingbias.com/2007/11/complex-wishes.html as an example,
a good data compressor will know what you mean when you say "get my mother out
of the burning building", in the sense that if
s1 = "'Do you mean to get her out alive?' 'Yes'"
s2 = "'Do you mean to get her out alive?' 'No'"
then string s1 has a higher probability, and therefore a shorter code length
than s2, and likewise for related questions with common sense answers, because
common sense knowledge improves compression of dialog with common sense
answers.
The recipe I proposed is still dangerous because (1) learning what our
coherent extrapolated volition is is a different problem than actually
motivating a system to grant our wishes according to this knowledge, and (2) a
sufficiently powerful and self improving compressor with the obvious utility
function could still turn the Earth into computronium.
-- Matt Mahoney, matmahoney@yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:01 MDT