From: Gordon Worley (redbird@rbisland.cx)
Date: Fri Feb 09 2001 - 18:59:59 MST
At 7:58 PM -0500 2/9/01, Eliezer S. Yudkowsky wrote:
>Gordon Worley wrote:
>>
>> When I'm uploaded, I
>> don't want to be forced to be Friendly and altruistic. If I choose
>> to be, that's one thing, but I should be able to keep on living as
>> selfishly as I choose.
>
>Samantha and I just ran through this one - the answer is the Sysop
>Scenario, an OS which permits the existence of malevolent
>superintelligences without risking the safety of any other citizens.
>Including the static uploads, the benevolent superintelligences, and the
>Amish on Earth. I can't imagine that any but the suicidal will accept
>unfiltered communications from you if you make yourself a malevolent SI,
>but that's the price you pay.
But what do you plan to do to control the SIs that are outside of
this OS? Sure, maybe 99% of the SIs will want to live in FriendlyOS,
but some will choose, when uploaded, to stay outside of the OS. Now
fine, those in FriendlyOS might choose to ignore such outsiders, but
they will not be able to ignore them if they did something in the
physical world. For example, let's say that all the SIs are busy
living in FriendlyOS, going about their simulations and enjoying the
altruism of the Sysop. Meanwhile, I and EvilJoe decide that we don't
want to be part of FriendlyOS, so we build our own, independent
computers to upload ourselves to. Then, after the brain scan, being
turned into code, and a few subjective centuries of self code
enhancement, both of us are equivilant technologically to anything
found in FriendlyOS. With a little more work, we get better, and
since the SIs inside FriendlyOS won't listen to us, they fail to
realize that we have become super powerful in the physical world.
One day, I'm doing some kind of post humanish thing, when suddendly
EvilJoe announces to the universe that he is going to destroy it.
I'm not ready for it, but that's my own fault. Those in FriendlyOS,
though, will have only the Sysop to blame. By letting them become
absorbed in their own world and by letting vis cycles go to being
altruistic towards SIs in the FriendlyOS, ve is unable to cope with
EvilJoe's actions. Thus, EvilJoe kills everything off to be made
into whatever ve desires, all because there weren't enough SIs aware
of what he was doing to stop him.
So, how do you propose to deal with this issue? One, I suppose,
would be to come up with some sort of legislation, but that could be
too easily avoided. Another might be to use force to jack everyone
into FriendlyOS, but the ethics of this are questionable. How are
you going to preserve Friendliness in a universe where any SI (and
not just ones from Earth, mind you) can exist outside and be
unFriendly?
-- Gordon Worley http://www.rbisland.cx/ mailto:redbird@rbisland.cx PGP: C462 FA84 B811 3501 9010 20D2 6EF3 77F7 BBD3 B003
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT