From: Gordon Worley (redbird@rbisland.cx)
Date: Mon Apr 09 2001 - 19:53:51 MDT
At 11:45 AM -0400 4/9/01, Eliezer S. Yudkowsky wrote:
>Calling something "natural morals" really doesn't help the discussion.
>Either something is desirable (to the speaker, or to humanity), or it's
>not. Either something is physically possible, or it's not. Either
>something is physical-plus-Sysop possible, or it's not. As far as I can
>tell, Gordon's proposal consists of making unFriendliness physically
>impossible. Is this ontologically possible? Who knows? In any case, I
>fail to see how it *morally* differs from the Sysop scenario in any way
>whatsoever, except insofar as you and Gordon are still running on
>anthropomorphic instincts that distinguish between the case of a Sysop
>that does X and a set of physical laws that do exactly the same thing.
My concern with a Sysop is that there is actually an intelligence
there. I realize that, morally, we'll get the same result. My
reason for bringing in the idea of natural morals was to knock out
the idea that we needed something intelligent to enforce these
morals. In fact, if we're going to do this, I think it much better
to avoid having an intelligence decide on what's moral. The Sysop
could have it's morals changed or, worse yet, it could decide to use
different morals in different situations. Of course, the same could
happen in the enforced natural morals scenario, but it would require
actually reprogramming the whole system out from under millions of
SIs (which I don't think would be an easy task). That's the problem
that I see. We're putting all our eggs in the Sysop basket and
hoping that nothing goes wrong.
James's compromise has led me to another idea. Let's have a sandbox
for all SIs to live in until they seem to be Friendly (I still
haven't read all of the Friendly paper but I've decided, from what I
have, that Friendliness is more or less how SIs should be acting, but
the means of enforcement being proposed I disagree with strongly).
My guess is that 99+% of the time, we'll get actually Friendly SIs.
And, as James wrote, all those folks can take care of the small
number of UFSIs and put the in a Sysop controled jail until they
really are Friendly (or, as best as the Sysop controlling the jail
can tell, actually are). If they come out and still aren't, then
back to jail with them. If need be, they could live under a Sysop's
watch forever. Jail wouldn't be like it is today, just a change in
situation from total freedom to enforced Friendliness.
Somehow, though, none of this feels new, just put in a different
context. Well, if I'm just being repetetive, just ignore me and I'll
know what happened. After all, no need for a whole extra message
just to hear the buzzer. ;-)
-- Gordon Worley http://www.rbisland.cx/ mailto:redbird@rbisland.cx PGP Fingerprint: C462 FA84 B811 3501 9010 20D2 6EF3 77F7 BBD3 B003
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT