Re: Recipe for CEV (was Re: Morality simulator)

From: Matt Mahoney (
Date: Sat Nov 24 2007 - 13:12:52 MST

Tom McCabe wrote:
> This is not a recipe for CEV, or for anything besides getting us all
> killed in nasty ways. See

I agree it will wipe out the human race, but I disagree that compression is
not CEV. CEV is a definition of friendliness, not a solution.

Using as an example,
a good data compressor will know what you mean when you say "get my mother out
of the burning building", in the sense that if

s1 = "'Do you mean to get her out alive?' 'Yes'"
s2 = "'Do you mean to get her out alive?' 'No'"

then string s1 has a higher probability, and therefore a shorter code length
than s2, and likewise for related questions with common sense answers, because
common sense knowledge improves compression of dialog with common sense

The recipe I proposed is still dangerous because (1) learning what our
coherent extrapolated volition is is a different problem than actually
motivating a system to grant our wishes according to this knowledge, and (2) a
sufficiently powerful and self improving compressor with the obvious utility
function could still turn the Earth into computronium.

-- Matt Mahoney,

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:01 MDT