From: Samantha Atkins (samantha@objectent.com)
Date: Mon Jan 29 2001 - 23:58:13 MST
"Eliezer S. Yudkowsky" wrote:
>
> Samantha Atkins wrote:
> >
> > But this are declaritive cognitive supergoals that you yourself, after a
> > lot of consideration, chose. Since you seem to be saying you will
> > establish Friendliness as the Supergoal in the SI by fiat, it is not the
> > same situation. In order to be fully intelligent an AI has to be able
> > to question its own goals and adjust as necessary. I don't see that you
> > can establish and hope to keep a Friendliness or any other supergoal for
> > all time without cripping the intellectual and self-evaluative power of
> > the AI.
>
> Um, when did I *ever*, *ever* say that I would establish Friendliness as
> the supergoal by fiat? For Friendliness to be stable, the AI needs
> human-equivalent or transhuman "strength of philosophical personality".
> The definition has to be supported by the same shapers that support it in
> me, at least as a starting point. Remember that programmer independence
> is a design goal, and that the AI learns about Friendliness by tracing out
> the causal chain that lies behind the volitional actions of the
> programmers. Eventually, Friendliness should be defined by reference to
> the interactional product of functional forces that are shared by all, or
> almost all, humans, at which point the AI or SI is programmer-independent.
>
> > > But I am *not* proposing to make AIs neurotic. "Friendliness is the only
> > > important thing in the world", or rather, "Friendliness is my goal
> > > system", is a perfectly healthy state of mind for an AI. And taking
> > > massive preventative action if Friendliness is threatened doesn't require
> > > a programmer assist; it's a natural consequence of the enormous unFriendly
> > > consequences of an AI community drifting away from Friendliness. I would
> > > *never* "make" an AI fear anything; at most, I would ask politely.
> >
> > I don't see this as a healthy state. It is an unexamined primary
> > belief. No, much stronger than that. It is the wired in basis of
> > everything else. Either it is forbidden and/or impossible to examine it
> > (by definition unhealthy) or it can be examined and found wanting. That
> > friendliness is desirable does not mean having it as an absolute is
> > either tenable or healthy.
>
> Yes, Friendliness can be examined and found wanting. This is a feature,
> not a bug. Nothing I said above is incompatible with that.
>
> > So is the idea of a super-SI that rules everything else on the basis of
> > a more or less hardwired and presumably immutable Friendliness
> > supergoal. Said SI will be most unfriendly to those entities that do
> > not agree with its notion of what friendliness entails.
>
> What? Why on Earth would ve care? Taken at face value, this statement
> sounds remarkably anthropomorphic. If you scream and hold protests about
> the evil dictator SI, the evil dictator SI will help you write your
> banners, organize the march, think of good slogans, and fulfill any other
> Sysop API requests that don't infringe on the living space of other
> citizens.
Please, do not bother to label my questions "anthropomorphic". Why
would ve care? Would ve care that some group was attempting to build
another SI of equal power? Would ve care if humans would not accept any
of its advice and insisted on being left severely alone? I am not
sure. So I am asking. Some things you have said lead me to believe
that there are definitely things the SI would be quite opposed to, even
violently so. Would you clarify what those are?
>
> > AFAIK it will
> > not necessarily be "friendly" to those entities that simply wish to
> > strike out on their own outisde of its influence.
>
> It's the Sysop's superintelligent decision as to whether letting someone
> Outside would pose an unacceptable risk to innocent sentients. My
> personal guess is that it does pose an unacceptable risk. If something
> doesn't pose an unacceptable risk to innocent sentients, you should be
> able to do it through the Sysop API. That's practically what Friendliness
What if onery sentiences simply do not want to have to pass all
decisions through this Sysop, no matter how intelligent and benign it
may be? This is not an unexpected situation. What will the Sysop do in
those cases? What if some group of sentients decided that what the
Sysop considered an unacceptable risk was perfectly acceptable to them?
Why would the Sysop want to forbid all entities that disagreed from
going somewhere outside its territory? Can't stand the possibility of
competition or that something might not be under its metaphorical
thumb? This brings up a question that I've never felt was adequately
answered. How is it good for humans, being just the type of onery
independent creatures that we are, to have a benign Sysop rule over us?
It strongly goes against the grain of the species. How will the Sysop
deal with the likely mass revolt? What will "Friendliness" dictate?
Simply wait it out as it holds all the cards? Will the Sysop be sure
this is actually being "Friendly" to the type of creatures we are? Are
you sure?
> *is*. If you want to play tourist in Betelguese, wrap a chunk of Sysop
> around yourself and take off. You won't be able to torture the primitives
> when you get there, but you'll be able to do anything else.
What if I simply want an extended vacation from Sysop controlled space?
>From what you have said, if I decide I want to extend that permanently
the Sysop will say no. Interesting. Do you honestly think humanity
will put up with this? Do you honestly think it is ok to effectively
force them to by leaving no alternative?
- samantha
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT