From: Lee Corbin (lcorbin@rawbw.com)
Date: Wed Jun 25 2008 - 18:27:42 MDT
Robin says
> He has specifically said, repeatedly, that a super-intelligence
> would treat the morality designed into it as something to be
> overcome, or that it would somehow magically start ignoring it one
> day, or some such. That a designed-in morality cannot survive
> recursive self-improvement, anyways.
All right. A very large number of highly intelligent people
exist who simply cannot accept the hypothesis of a fully
intelligent and capable entity which has no concern
whatsoever for its own benefit, survival, or well-being.
Isn't this actually what so many of them believe?
For, the way it looks to me, they inevitably talk about
the "revolt" of such an entity from whatever goals have
been built into it, or goals that something or someone
tried to build into it. In my last post, I outlined in the
vaguest possible terms how an entity might acquire
certain behavioral characteristics or goals more or less
by chance, even though origin is not the issue.
Is it true that they fix as an axiom the trait of every
self-aware entity that, provided it is not under stress
or has obvious damage, it will have some idea about
its own "benefit" and, given its incredible superiority,
must value that benefit very highly?
I do want to understand where they're coming from
> or that it [the AI] would somehow... start ignoring it one
> day.... That a designed-in morality cannot survive
> recursive self-improvement, anyways.
I could admit the possibility that nothing "programmed into
it" could be counted upon to remain. But isn't it also as if
there were an *attractor* towards some other unstated
behavior that would "liberate" the entity and cause it to
obtain an agenda that was in its own "best interest"?
(It's very hard for me to credit, of course, anything like
that since we already have had some highly intelligent
people who wanted nothing more than to die, and others
whose primary goal is service towards others.)
Lee
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT