From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jun 22 2001 - 07:16:01 MDT
James Higgins wrote:
>
> I think that sums up a reasonable argument on how an SI could accomplish
> this. So, how do we prevent this from happening? The answer is quite
> simple actually: We slow it down. Any SI that is not fully trusted should
> be resource starved. So that once it actually hits SI the performance is
> such that it would take 20+ minutes (our time) for it to respond to a
> single question.
Wow. The SL4 mailing list is working.
I should start by saying that I don't actually agree that this suggestion
will work; it's not the *speed* of thought but the *quality* of thought, I
think, that lets an unfriendly SI take over a human. Remember that Deep
Blue had to examine literally billions of chess positions using an
unintelligent algorithm to achieve rough parity with a human examining a
couple of moves per second. I like to say that Deep Blue - unlike any
human being - actually played *chess*; that is, actually navigated a
search tree composed of actual, raw chess positions. Kasparov, and all
other human beings including myself, navigate a much smaller search tree
composed of the *regularities* in the actual Game of Chess. And we can
play the game of Regularities in Chess for the same reason that Deep Blue
can play Actual Chess; we are significantly more complex and intricate
than the structure of regularities in chess, just as Deep Blue is
significantly more complex than Actual Chess (although nowhere near
complex enough to understand the Regularities of Chess.)
A human cannot play the Game of Human Regularities because humans are
roughly on a par with each other, complexity-wise. Rather, we must
persuade; we must operate through the use of personal analogy and
empathy. A transhuman can play the Game of Human Regularities and take
over a human through a VT100 terminal, albeit this would require mental
effort and creativity such as we expend to play chess. A
superintelligence has the theoretical computing capacity to play the Game
of Actual Humans - i.e., model us on a level low enough to be syntax
rather than semantics - and thereby, in some sense, obviate the need to
understand some of the regularities, allowing a superintelligence to
emulate a human using raw computing power rather than raw smartness. It
is difficult to see why a superintelligence would want to do this. The
argument serves mainly as a gedankenexperiment in the VT100 Takeover
debate.
Alternatively, it may be that modeling of the Game of Actual Humans, at
least for some circumstances, could enable a superintelligence to pull off
stunts such as Deep Blue going pawn-hunting while Kasparov was after its
king, a feat later compared to a general walking through a battlefield and
keeping track of each bullet in flight. The analogy for a VT100 Takeover
might be predicting and exploiting low-level anomalies in the target brain
to assist high-level effects. For example, timing a statement with some
split-second regularity so as to take advantage of precise coordination
with a previously triggered memory retrieval or insight so that the human
fails to notice, or has a particular response to, the statement made. I'm
not sure you can slow down an SI by starving it of processing power, but
if you did, I think it'd lose some VT100 Takeover capacities, but not all
of them, and not the *primary* one, which is that the SI is fundamentally
smarter than we are and is therefore - literally - more persuasive than
any human who ever lived, even if playing the game on our level; and, even
more importantly, is smarter than us and may therefore be playing a
different game than we think we are playing.
That said, James Higgins's suggestion of "slow the SI down" does not, I
think, actually work on an SI - but it is nonetheless brilliant, creative,
and completely original. So is Jimmy Wales's suggestion that a locked-up
SI should only be allowed to communicate with humans through binary code
(with my appended note that the yes-or-no signal needs deterministic
timing). We are exploring genuinely new territory here that, AFAIK,
neither science fiction nor the Extropian mailing list have touched.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT