From: James Higgins (jameshiggins@earthlink.net)
Date: Fri Jun 22 2001 - 19:51:30 MDT
First, thanks for the compliment.
Second, a little clarification on my intent. I realize that the quality of
thought is very important, but we are not in a position to understand what
better thought is like. However, we can comprehend what faster thought is
like so I used that for my example. Further, any statement that an SI
could take over a human via VT100 using some kind of "magic" is hard to
swallow. It may be possible, but it is unlikely that we will really know
until it is too late. My goal was to demonstrate a straight forward method
by which an SI could feasibly control a human that was comprehensible and
believable. Hopefully I accomplished what I set out to do.
Lastly, on the issue of speed. I believe that slowing down an SI
sufficiently would have a serious effect on its ability to persuade us. If
it is only able to converse at the rate of approximately 1 sentence per
half hour, I would imagine that all timing attacks are out of the
question. Further, better thought may not be able to overcome faster
thought in many situations. Lets look at an example. You have a very well
trained dog named Rover who can fetch the newspaper and is very compliant
when given an instruction. Then, lets say that you are suddenly slowed
down to approximately 1/1000ths your current speed. You are still very
much smarter than Rover, but it becomes incredibly hard to get rover to do
anything by command. Even if your output came out in realtime (buffered),
you would actually start issuing the "Fetch the paper" command 30 minutes
before it was delivered. By that time Rover could be in a different room
or chewing his favorite bone and thus not paying attention to you. It
would be more luck than anything if you did manage to get Rover to fetch
the bone.
Also, I have a question. Does anyone plan to give an unknown SI detailed
information about human anatomy, brain function and/or psychology? Somehow
I doubt that an SI will have access to this type of information and I find
it incredibly hard to believe that it could pull off any sort of timing or
low-level attack without this data.
At 09:16 AM 6/22/2001 -0400, Eliezer wrote:
>James Higgins wrote:
> >
> > I think that sums up a reasonable argument on how an SI could accomplish
> > this. So, how do we prevent this from happening? The answer is quite
> > simple actually: We slow it down. Any SI that is not fully trusted should
> > be resource starved. So that once it actually hits SI the performance is
> > such that it would take 20+ minutes (our time) for it to respond to a
> > single question.
>
>Wow. The SL4 mailing list is working.
>
>I should start by saying that I don't actually agree that this suggestion
>will work; it's not the *speed* of thought but the *quality* of thought, I
>think, that lets an unfriendly SI take over a human. Remember that Deep
>Blue had to examine literally billions of chess positions using an
>unintelligent algorithm to achieve rough parity with a human examining a
>couple of moves per second. I like to say that Deep Blue - unlike any
>human being - actually played *chess*; that is, actually navigated a
>search tree composed of actual, raw chess positions. Kasparov, and all
>other human beings including myself, navigate a much smaller search tree
>composed of the *regularities* in the actual Game of Chess. And we can
>play the game of Regularities in Chess for the same reason that Deep Blue
>can play Actual Chess; we are significantly more complex and intricate
>than the structure of regularities in chess, just as Deep Blue is
>significantly more complex than Actual Chess (although nowhere near
>complex enough to understand the Regularities of Chess.)
>
>A human cannot play the Game of Human Regularities because humans are
>roughly on a par with each other, complexity-wise. Rather, we must
>persuade; we must operate through the use of personal analogy and
>empathy. A transhuman can play the Game of Human Regularities and take
>over a human through a VT100 terminal, albeit this would require mental
>effort and creativity such as we expend to play chess. A
>superintelligence has the theoretical computing capacity to play the Game
>of Actual Humans - i.e., model us on a level low enough to be syntax
>rather than semantics - and thereby, in some sense, obviate the need to
>understand some of the regularities, allowing a superintelligence to
>emulate a human using raw computing power rather than raw smartness. It
>is difficult to see why a superintelligence would want to do this. The
>argument serves mainly as a gedankenexperiment in the VT100 Takeover
>debate.
>
>Alternatively, it may be that modeling of the Game of Actual Humans, at
>least for some circumstances, could enable a superintelligence to pull off
>stunts such as Deep Blue going pawn-hunting while Kasparov was after its
>king, a feat later compared to a general walking through a battlefield and
>keeping track of each bullet in flight. The analogy for a VT100 Takeover
>might be predicting and exploiting low-level anomalies in the target brain
>to assist high-level effects. For example, timing a statement with some
>split-second regularity so as to take advantage of precise coordination
>with a previously triggered memory retrieval or insight so that the human
>fails to notice, or has a particular response to, the statement made. I'm
>not sure you can slow down an SI by starving it of processing power, but
>if you did, I think it'd lose some VT100 Takeover capacities, but not all
>of them, and not the *primary* one, which is that the SI is fundamentally
>smarter than we are and is therefore - literally - more persuasive than
>any human who ever lived, even if playing the game on our level; and, even
>more importantly, is smarter than us and may therefore be playing a
>different game than we think we are playing.
>
>That said, James Higgins's suggestion of "slow the SI down" does not, I
>think, actually work on an SI - but it is nonetheless brilliant, creative,
>and completely original. So is Jimmy Wales's suggestion that a locked-up
>SI should only be allowed to communicate with humans through binary code
>(with my appended note that the yes-or-no signal needs deterministic
>timing). We are exploring genuinely new territory here that, AFAIK,
>neither science fiction nor the Extropian mailing list have touched.
>
>-- -- -- -- --
>Eliezer S. Yudkowsky http://intelligence.org/
>Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT