From: James Higgins (jameshiggins@earthlink.net)
Date: Mon Apr 09 2001 - 20:24:10 MDT
At 11:45 AM 4/9/2001 -0400, you wrote:
>James Higgins wrote:
> >
> > Ah, I may have an interesting compromise. I do agree that us mere mortals
> > could use something and would be helpless against SIs. How about if the
> > sysop ONLY enforced happiness towards humans. Once uploaded, SIs would be
> > free to do as they pleased, as long as they didn't interact with
> > humans. When they did interact with humans, they would be forced to follow
> > a set of friendly rules. Humans would also be subject to these friendly
> > rules. Eliezer, what do you think about this?
>
>This makes sense if, and only if, superintelligent entities are knowably
>Friendly. If I recall correctly, that's been your premise from the
>beginning.
Well, I have mentioned that before as a possibility. But I'm not locked
into any particular belief since it is way too early to understand the
nature of the singularity.
However, I disagree with your requirement that all entities must be
knowably friendly. Why is this? What happens if this is not the case?
>Calling something "natural morals" really doesn't help the discussion.
>Either something is desirable (to the speaker, or to humanity), or it's
>not. Either something is physically possible, or it's not. Either
>something is physical-plus-Sysop possible, or it's not. As far as I can
>tell, Gordon's proposal consists of making unFriendliness physically
>impossible. Is this ontologically possible? Who knows? In any case, I
>fail to see how it *morally* differs from the Sysop scenario in any way
>whatsoever, except insofar as you and Gordon are still running on
>anthropomorphic instincts that distinguish between the case of a Sysop
>that does X and a set of physical laws that do exactly the same thing.
I think you are getting Gordon and myself mixed up here. I don't agree
with what I believe to be his viewpoint on morals in this context. What I
believe we do both agree on is that a sysop is not necessarily a good idea.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT