Re: Learning to be evil

From: Gordon Worley (
Date: Sat Feb 10 2001 - 16:51:40 MST

At 2:04 AM -0500 2/10/01, Eliezer S. Yudkowsky wrote:
>Gordon Worley wrote:
>> But what do you plan to do to control the SIs that are outside of
>> this OS?
>We *just* went through all this. *Please* read the "Beyond evolution"

I have and am not satisfied. To quote how you left the matter:

> No. Others can go outside who may have more nefarious motives. Are you
> claiming they would never tire of being bloody tyrants, never feel
> remorse, never seek to undo some part of the painful ugly creatin they
> made?

Some would, some wouldn't.

What kind of answer is this supposed to be? Your answer is that some
will step outside of the FriendlyOS or not enter at all and be evil,
possibly to the point where they destory everything else (or maybe
even themselves; see the scenario in my previous post), and that's
how it is? If there were no FriendlyOS, everyone might have to keep
worrying about everyone else, but at least it would make sure that no
one could do something without at least the possibility of noticing
their actions, whereas everyone in the FriendlyOS will be so busy in
their VRs not to even care about physical reality. The Sysop may
have a lot of power, but ve is just a finite entity like the rest of
us, even with the Singularity, and as such at some point can't do any
more, inside or (more importantly) outside of the FriendlyOS.

Samantha tries to offer a response, but it just assumes that SIs get
tired of something. But, depending on programming, this might never

Gordon Worley
PGP:  C462 FA84 B811 3501 9010  20D2 6EF3 77F7 BBD3 B003

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT