From: rolf nelson (rolf.hrld.nelson@gmail.com)
Date: Sun Aug 26 2007 - 11:07:11 MDT
On 8/26/07, Gwern Branwen <gwern0@gmail.com> wrote:
> A few thoughts: if a UFAI's supergoal ineluctably depends on it being in the 'real' universe, on not accomplishing whatever it is in the simulation but in reality, then wouldn't this motivate it to destroy the human race as soon as efficient?
In the given scenario, the UFAI is indifferent to anything relating to
UFAI* in that case. If you move beyond this proof-of-concept "toy
universe" to a UFAI with a more complicated utility function that
encompasses multiple supergoals, and one of the supergoals is "if my
task is impossible, I want to know right away" then yes, ending the
simulation may paradoxically be a reward rather than a punishment.
Beyond the proof-of-concept, the general concept then is to reward
UFAI for good deeds and/or punish UFAI for bad deeds.
> And how could one construct the threat such that the UFAI cannot rationally claim to believe that the simulation threat is a bluff?
>From the FAI's point of view, running UFAI* is, indeed, a pointless
waste of time. However, the following conditions would probably be
sufficient for FAI to run UFAI*:
1. A pre-singularity human promised to ask FAI to run UFAI*;
2. Post-singularity, instead of reneging, the human asks FAI to run
UFAI* (because humans are stubborn and illogical, or because there's
social value to be gained by being perceived as "someone who keeps his
promises");
3. Running UFAI* requires an insignificant amount of FAI's resources; and,
4. Human autonomy is one of the things the FAI values in its utility function.
Beyond the proof-of-concept, instead of saying "FAI" you could more
generally say "any Strong AI that values human autonomy enough to
assist me in accomplishing my quirky little irrational goals", or "any
Strong AI that would be willing to run UFAI* if I ask nicely".
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT