Re: ESSAY: How to deter a rogue AI by using your first-mover advantage

From: Norman Noman (
Date: Wed Aug 29 2007 - 04:01:34 MDT

> > And even if you're RIGHT, and there is a pandemonium of human infighting
> via
> > simulation which cancels out to nothing, there is no reason rolf's plan
> > cannot be implemented as well.
> It wouldn't do any harm to implement it, but to be consistent an agent
> should take any threat via simulation from the next level up
> seriously.

Your strategy of ignoring the problem may be consistent, but it's hardly
rational. ou seem to believe that by providing hypothetical (and in my
opinion quite contrived) scenarios in which simulation traps result in
unpleasant or confusing situations, you've given adequate grounds for
plugging your ears and going la-la-la, as though being rational should
always be easy and simple.

If there is only one known such threat it is easy to decide
> what you should do, but the moment it becomes clear that it is a cheap
> strategy to gain advantage everyone will jump on the bandwagon.

It is not a cheap strategy to gain advantage, it is a rather elaborate
strategy to encourage cooperation between entities which otherwise have no

> If both parties run X simulations, your likelihood of being in one of A's
> > simulations rather than B's simulations is proportional to the
> likelihood of
> > A existing in the first place rather than B. As X goes to infinity, this
> > likelihood does not change.
> Right, but if there are many competing interests rather than just two,
> it will be difficult to choose between them. You would have to try to
> guess which organisation would be the most determined and the most
> capable of running their proposed simulation in the long term.

Which according to you is impossible for some hand-wavey reason.

I think what bothers people about the simulation argument is that it seems
so akin to religion, or at least to a certain very defensive religious
position, that god created the world, fossils, etc. all to look exactly as
though they were created naturally. The argument is consistent, but very
weak because it fails to provide a motive for god or a reason for his

The simulated karma trap gets to this scenario from the other direction, and
the reason it's worth bothering about is that it DOES provide a motive and
it DOES provide a reason for the existence of the creator. Indeed, it is our
motives and our existence.

It seems to me you have reasoned by analogy that if something smells like
religion, it must be unsound, and then attempted to rationalize your
position by saying the math is so confusing we ought to ignore it.

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT