From: Tennessee Leeuwenburg (hamptonite@gmail.com)
Date: Wed Jul 13 2005 - 23:07:46 MDT
On 7/14/05, Eliezer S. Yudkowsky <sentience@pobox.com> wrote:
> Tennessee Leeuwenburg wrote:
> >
> > I suppose I'm unwilling to accept the paperclips position to some
> > extent, for a variety of reasons.
> >
> > Is a truly intelligent AI ever going to make the kind of monumental
> > slip-up required to decide to do something so blatantly dumb as just
> > cover the universe in paperclips?
>
> "Giant bug or
> not, it's likely to be wired pretty much the same way we are, so presumably it
> will also find human females sexually attractive."
<snip>
> Paperclips are not inherently a dumb idea.
Point. But nonetheless, it seems to be a cop-out to suggest that we
can know _nothing_ about the minds of nonhuman intelligences and/or
super-intelligences.
Before leaping from one example of someone who failed to consider that
a different kind of mind would be, well, different, we should try to
establish some bounds on the problem.
We have access to consciousness through introspection. Can we identify
which elements of consciousness are arbitrary, and which are not? To
put it another way - can we identify which elements of ourselves might
be preserved, or perhaps even necessarily must be preserved, in
another kind of mind.
Is emotion, for example, a natural byproduct of the combination of
intelligence, consciousness and experience? Perhaps it is not - but
perhaps there are some identifiable examples.
What do you think? Do you think that what we have access to as
intelligent beings is not even the same kind of thing that another
intelligence might have access to? Does it make sense to call FAI
intelligent if the mind cannot be in any sensible way called "the
same"?
> > I know people have posed race conditions between FAI and paperclips,
> > but there seems to me to be a kind of contradiction inherent in any AI
> > which is intelligent enough to achieve one of these worst-case
> > outcomes, but is still capable of making stupid mistakes.
>
> Actions are only "stupid mistakes" relative to a cognitive reference frame.
Partial point. Obviously irrationality is irrationality however you
slice it. I don't care who you are, (A -> B) -> (!B -> !A) is going to
stay a logical law. But you rightly didn't respond like that, my point
was the "stupidity", the disconnect between the logic and the sense if
you will, of being able to formulate paperclips as a goal, and
believing it to be a good idea.
One thing about humans - we ask existential questions. Why are we
here? What shall we do now? In a sense, human intelligence is a defeat
of instinct and of mindless goal-following. A superintelligence poses
a strange reversal. By being able to remove uncertainty, it is
supposed that in answering the questions "Why am I here?" and "What
shall I do now?", the AI will return to a near-instinctual level of
behaviour, with deadly efficiency if it so chooses.
Do you think that's a fair analysis? Some kinds of AI are scary
because they might come to a doubt-free conclusion?
Or do you perhaps not think that existential questions are like that.
It might be that the greater the intelligence, the less cosmic
certainty one has. We should have the opportunity to interrogate
merely somewhat superintelligent beings about this question at some
point before singularity.
> > Does it make sense that something so intelligent could have such mindless goals?
>
> 3.85 billion years of natural selection was powerful enough to poof primordial
> soup into peregrine falcons, a nontrivial amount of design ability. Yet
> natural selection's goals were pretty mindless from a human standpoint, just
> as human beings having sex with contraception is pretty stupid from the
> mindless standpoint of natural selection.
>
> You can define an UFAI as "unintelligent" but it will still have a
> self-consistent goal system that can tear apart stars.
Yep.
> > I'm also willing to accept the risk-in-principle posed by advanced
> > nanotech, or some kind of "subverted" power which destroys humanity,
> > but I'm both reluctant to tag it as truly intelligent, and also
> > doubtful about the real possiblity.
>
> It doesn't matter what you call a real thing. Definitions don't control
> physical outcomes.
I thought that was the point I was making. It's worth considering the
in-principle risk of such a situation, but just because we can imagine
it, doesn't make it physically possible.
> As for the real possibility, a paperclip utility function looks perfectly
> self-consistent to me. If anything it looks a lot more self-consistent than a
> human.
Yep.
> > To some extent, there is a trade-off between efficiency and efficacy.
> > For example, the energy requirements might be too high to sustain
> > existence across the void of space. Just as lions in the sahara starve
> > when there is no food, so being powerful is not always a survival
> > advantage. I'm sure this point may have been up before, but I don't
> > know that it's a given that evil nanotech
>
> Evil nanotech? Evil is a term that applies to cognitive systems, not systems
> of a particular size. And an UFAI is not evil; it just transforms local
> reality according to a decision criterion that makes no internal mention of
> human life.
I was just grabbing some syntactic efficiency. There was no need to
miss my point. That point was that there's a constraint problem here,
and that some of the imagined worst-case scenarios may turn out not to
be possible.
Cheers,
-T
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:51 MDT