From: Richard Loosemore (rpwl@lightlink.com)
Date: Wed Aug 17 2005 - 16:59:54 MDT
This hypothetical paperclip monster is being used in ways that are
incoherent, which interferes with the clarity of our arguments.
Hypothesis: There is a GAI that is obsessed with turning the universe
into paperclips, to the exclusion of all other goals.
It is supposed to be so obsessed that it cannot even conceive of other
goals, or it cannot understand them, or it is too busy to stop and think
of them, or maybe it is incapable of even representing anything except
the task of paperclipization...... or something like that.
Anyhow, the obsession is so complete that the paperclip monster is
somehow exempt from the constraints that might apply to a less
monomaniacal AI. And for this reason, the concept of a paperclip
monster is used as a counterexample to various arguments.
I submit that the concept is grossly inconsistent. If it is a *general*
AI, it must have a flexible, adaptive representation system that lets it
model all kinds of things in the universe, including itself.
[Aside: AI systems that do not have that general ability may be able to
do better than us in a narrow area of expertise (Deep Thought, for
example), but they are incapable of showing general intelligence].
But whenever the Paperclip Monster is cited, it comes across as too dumb
to be a GAI ... the very characteristics that make it useful in
demolishing arguments are implicitly reducing it back down to sub-GAI
status. It knows nothing of other goals? Then how does it outsmart a
GAI that does know such things?
Or: it is so obsessed with paperclipization that it cannot represent and
perceive the presence of a human that is walking up to its power socket
and is right now pulling the plug on it .....? I'm sure none of the
paperclip monster supporters would concede that scenario: they would
claim that the monster does represent the approaching human because the
human is suddenly relevant (it is threatening to terminate the Holy
Purpose), so it deals with the threat.
I agree, it would understand the human, it would not be so dumb as to
mistake the intentions of the human ... because it *does* have general
intelligence, and it *does* have the ability to represent things like
the intentions of other sentients, and it *does* spend some time
cogitating about such matters as intention and motivation, both in other
sentients and in itself, and it does perceive within itself a strong
compulsion to make paperclips, and it does understand the fact that this
compulsion is somewhat arbitrary .... and so on.
Nobody can posit things like general intelligence in a paperclip monster
(because it really needs that if it is to be effective and dangerous),
and then at the same time pretend that for some reason it never gets
around to thinking about the motivational issues that I have been
raising recently.
That is what I meant by saying that the monster is having its cake and
eating it.
I see this as a symptom of a larger confusion: when speculating about
various kinds of AI, we sometimes make the mistake of positing general
intelligence, and then selectively withdrawing that intelligence in
specific scenarios, as it suits us, to demonstrate this or that failing
or danger, or whatever.
I am not saying that anyone is doing this deliberately or deceiptfully,
of course, just that we have to be vary wary of that trap, because it is
an easy mistake to make, and sometimes it is very subtle. I have been
attacking it, in this post, in the case of the paperclip monster, but I
have also been trying to show that it occurs in other situations (like
when we try to decide whether the GAI is being *subject* to a drive
coming from its motivation or is *thinking about* a drive that it
experiences.).
Does anyone else except me understand what I am driving at here?
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:23:01 MST