Re: ESSAY: How to deter a rogue AI by using your first-mover advantage

From: Norman Noman (
Date: Thu Aug 23 2007 - 08:02:58 MDT

On 8/22/07, Gwern Branwen <> wrote:
> Obviously, if your design is very unusual and odd and haphazard, then it'd
> be easier to conclude that you are not in the karma simulator - who on earth
> would want to engage in expensive testing of weird and unlikely designs?

 It's important to bear in mind that the FAI is simulating RAIs because by
doing so, it ensures that real RAIs would know it would, and thus have some
incentive to act cooperatively.

It is not simulating them to test them, or to observe them in any way, but
simply so that they exist. This is a form of a newcomb's paradox.

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT