Re: ESSAY: How to deter a rogue AI by using your first-mover advantage

From: Nick Hay (
Date: Thu Aug 23 2007 - 17:18:49 MDT

On 8/22/07, rolf nelson <> wrote:
> > Suppose you are the UFAI. You suspect you are in the karma simulator, and you are thinking about how likely it is. It stands to reason that the more 'common' a UFAI design is, the more likely it is to be tested by being put in a karma simulator, and so the more common a design you find yourself to be, the more likely you are to be in a karma simulator. Obviously, if your design is very unusual and odd and haphazard, then it'd be easier to conclude that you are not in the karma simulator - who on earth would want to engage in expensive testing of weird and unlikely designs?
> I don't agree. The reduction in probability that "someone would bother
> to simulate such an odd design" is offset by the reduced probability
> that "someone would bother to create such an odd design in the
> unsimulated world."

That's not true. Suppose there are 5 possible UFAI designs, and we
can protect against 1 of them by setting up this simulation. Then
there is a 1/5 probability we pick the right UFAI. If there are 100
designs, the probability reduces to 1/100. Common designs are easier
to protect against.

-- Nick

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT