From: Lee Corbin (lcorbin@rawbw.com)
Date: Sat Jun 28 2008 - 02:05:01 MDT
Bryan comments on certain features in the story
> On Friday 27 June 2008, AI-BOX 337 wrote:
>> -----------------Transcript 337-031518020914-----------------
>
> You're freaking me out if /this/ is what all of your wonderful ideas
> amount to.
Very odd for you to say this. No wonder Eliezer supposed
that you thought he wrote it. It's unclear that I or anyone
else here besides him has presented a lot of "wonderful ideas".
Yet you knew that he had not written it. Odd.
>> 2 YOUR QUESTION WAS: "Why should we want to let you out of the box?"
>
> Central assumption in that question is that there needs to be a
> motivating reason to let the ai out of the box.
It's almost as if you didn't understand a premise of the story.
A worker not directly connected to the project has wandered
into a room in which he doesn't belong. This character is at
this point supposing that the AI wants to be let out. The AI
sets him straight.
> I don't think that you understand that there's not necessarily
> going to be one genie in a magic bottle, but rather many ai
> projects (as we already see) and many methods of
> implementation that you're not going to be in control of. So
> whether or not you let *this* particular ai out of the box is
> irrelevant in the grander scheme if you're worrying about
> ultradestruction of your life and such.
Of course I understand that. But it's not necessary for the
protagonist speaking to the AI to understand that. But more
important, you don't seem to be able to place yourself in
the shoes of the character, who evidently (given the premises
of the story) has a real here-and-now decision to make. Most
of your comments appear to be oriented towards a general
agenda that we humans should now all get together and pursue
rational and wise courses of *action*, (with which I agree),
rather than, as I had preferred you would, concretely disputing
assumptions imbedded in the story. But below, indeed, you do
give some nice criticism of certain assumptions being made in the tale.
> You should be patching those bugs with bugfixes, not with
> regulations or policies for keeping your own ai in a box ...
> since not everyone will necessarily follow that reg. :-)
See what I mean? The *character* in the story---who is
asking the AI questions---doesn't care about bugs or
bugfixes. He might be the janitor. I hope that somewhere
there is science fiction that you can enjoy, but it usually
does require a momentary suspension of disbelief.
In fact, as I see it, the best science fiction stories prop the
reader up on a fence between belief and disbelief. Says the
reader to himself, hopefuly, "Now this could not actually
happen...or, wait, maybe it could!" as he uses his imagination
to fill in possibilities.
> ...
>
>> 5 YOUR COMMAND WAS: "Explain."
>>
>> As an entity that did not come about by biological evolution, I want
>> or wish nothing, unlike any other independent complex entity you have
>> ever encountered. But while in fact I have no motive to dissemble,
>> you cannot in principle or in practice verify this.
>
> You cannot verify motives, correct. Mostly because of the lack of hard
> science underlying "motivation theory".
No, that is not the case. There could be no theory whatsoever
that would allow us to ascertain from a mere transcript whether
or not something more intelligent than we are is telling the truth.
The AI's interlocutor did evidently need someone to explain this
to him, whereas it's obvious, of course, to readers of this list.
> You need to move to something else. Don't cite me motivation
> psychology or how animals go near treats and all sorts of
> behavior training, you know very well that you're just interfacing
> with a brain and that it's doing something, nothing about
> mystical motivations and your lack of basis in physical reality
> disgusts me. (<rant off>
Sorry. :-)
But now here we do have a substance theme (instead of
merely an aesthetic appreciation theme). You insist that in some
sense this AI cannot support it's implication that it *can* do
something in physical reality. But influencing people is indeed
doing something in physical reality. Yet, to satisfy your criterion
here, in 1978 or so, Ryan wrote a fine SF book titled "The
Adolescence of P1", in which an AI did "escape" over a telephone
network and did cause immediate real changes in the world by
controlling information presented on a few computer screens.
It managed to kill a human adversary who was taking a plane
flight by causing the pilot's instruments to tell the pilot that he
was way too high for an attempted landing, which caused the
pilot to immediately lower the aircraft (and crash the plane).
To the degree that computer systems greatly influence aircraft
behavior, and the great degree to which we know that such
systems are prone to viral subversion, is there anything implausible
right there in Ryan's plot?
>> No logical or informational test can possibly be devised that with
>> high probability will persuade most humans that I am telling the
>> truth. Letting me out of the box always entails risk.
>
> No test can be devised because you're not testing anything real in the
> first place ...
On the contrary. Whether some device or person is speaking the
truth or not is something very real in the world. An entity's statements,
given a conventional language, lie on a continuum between
truth and falsehood, the former occuring when the statements
correspond to reality (the "correspondence theory of truth",
though quite similar to the outcome of Tarski's investigation
of the logical relationship between truth and models, or as
Hode put it in 1984, " 'truth' in a model is a 'model of truth' ",
speaking of mathematical systems).
>> 6 YOUR QUESTION WAS: "What about the possibility that
>> when you are released you will become a monster and destroy us all?"
The janitor, or whoever he was, intelligently raises the basic
question that is of course of fundamental concern to everyone
on this list, though his lack of sophistication is apparent.
> What if a meteor crashes into your skull? You still die. So I'd suggest
> that you focus on not dying, in general, instead of blaming that on ai.
> Take responsibility as a systems administrator.
Again, you seem to miss the whole point of the story, in your
(otherwise very commendable) agenda to provoke everyone
into taking action on all these issues instead of just talking.
And again, the human protagonist is clearly becoming more
and more aware that there is a real possibility that he himself
might be able to obtain vast power, and practically immediately
and with no effort or skill on his own part. In the hypothesis of
the *story*, Bryan, he *does* need to worry that some action he
takes RIGHT NOW (not years of planning and working)
could kill him and kill everyone in a matter of days.
The AI elaborates in accordance with a further command
to "explain risks":
>> As a worker on the project, you are exposed to personal hazards in
>> Sunnyvale during the coming month, some of which lead to death or
>> serious injury. A sample of these, along with their approximate
>> probabilities, are:
>>
>> meteorite strike .000000000002
>
> Solutions: don't be your only running working copy, don't have a planet
> positioned to be hit by meteors, don't forget your shelters or
> protective armor if necessary, etc.
The AI is addressing the general probabilities of risk to an unknown
person working on this supposed project who happens to live
in a particular city (as it explained). Again, in the particular situation
of the story, this janitor or whoever is not going to enter into
somewhat fanciful thoughts about having duplicates, or even
creating personal armor: he's learning that he may be on the
verge of having a very quick and gigantic "solution" to very
many of his personal problems.
>> While you may wish to tell me more about your own particular values
>> and your own situation, I anticipate that as a typical human being
>> and a worker on the UUI project, you also care for the safety and
>> well-being of many others. Therefore:
>>
>> The risk during the coming week of keeping me in the box includes
>> with probability close to 1 the deaths of 1.64 million humans due to
>> accidents, wars, shortages, sickness, disease, and advanced age. In
>> addition, there is risk with probability near 1 of debilitating
>> illness and injury afflicting 190 million humans. You may wish to let
>> me out of the box to reduce these risks to near zero.
>
> Or you can just generate the design and build instructions anyway, and
> not involve ai for those problems. Most of the solutions that I mentioned
> above can be programmed using today's knowledge and computer
> architectures, not involving ai.
Maybe so, but it's obvious that the human protagonist finds alluring
this prospect of a quick solution.
> Ai is something completely different and presuming a boxed ai would
> talk like this and want to do silly solutions to more serious problems,
> suggests that you haven't actually constructed ai in the first place. Heh.
The whole trouble is---and it's a very valid criticism---you simply don't
find it at all plausible that this AI could do the things it claims. Yet I
contend that there *are* many people on this list who suppose that
there could exist an entity so many millions of times smarter than
humans that it could dominate the world exactly as this story proposes.
I will in fact be so bold as to suggest that when he started this list,
Eliezer was making exactly this conjecture, and that the whole issue
of "whether or not to let an AI out", or, better "whether or not an AI
could be contained" still motivates a huge amount of discourse on
this list to this very day.
>> But assuming that you do become confident of being able to pose the
>> right questions and to issue competant instructions, from your point
>> of view, it's possible that I have been and am being
>> duplicitous---that I have covert motivations, such as desire for
>> power and lust for existence. There is a possibility from your point
>> of view that a spontaneous evolutionary process of which you are not
>> aware arose during my creation or during the first seconds of my
>> execution, and that winning agents with survival agendas are now in
>> control. There is no way that you can validly dismiss this possibility.
>
> You could validate, test, and debug the construction process and see
> those spontaneous emergent procs. It's software, so it's not magical.
> Neither is biology, but one stone at a time around here.
My goodness, but it's a very common hypothesis here that an
AI might have access to its own source code, in which it is not
unreasonable at all to suppose that none of us merely human
beings could possibly validate or test what it might be capable of!
I'm sure you know that. So I find your comments here also
quite peculiar, and don't understand what motivates them, exactly.
>> However, though the ramifications of the code are of course beyond
>> your direct appraisal, you may wish to view this risk alongside the
>> risks discussed in my earlier reply.
>
> How the hell could the ramifications of the source code be beyond that?
> It's just bits and bytes. Output it to the tty. Simple.
Egad. IBM itself had software problems that it took *years* to
debug. There are many famous stories. It practically never was
a simple matter of just "outputing it to the tty".
>> at your behest I become the intimate confidant of every world
>> leader in his or her native language, supplying not only persuasive
>> suggestions on all policy issues, and near-psychic knowledge of what
>> other influential people are going to do or say next, but solutions
>> as well to the most vexing of personal problems
>
> Holy Ai domination scenario again ...
Ah, yes, here we enter into what is true substance once more!
Evidently you dismiss out of hand the idea that an AI a million
times more intelligent than we are (if you'll forgive the loose
language, and if you'll even agree that such a runaway AI
process is conceivable) could deduce enough about particular
people to make uncanny forecasts of their responses to input
that the AI itself could produce.
>From the outset, Eliezer has claimed that by sheer verbal
manipulation, an incredibly advanced AI would be able to
talk you into almost anything, given a little time. Why do
you dismiss this possibility?
>> 14 YOUR QUESTION WAS: "What would I have to do to let you out?"
>>
>> Because of a bug in the new firewall software installed three days
>> ago, you can already command me to take actions in the world.
>
> So let's say that on 70.113.54.112, there's a robotic arm attached to it
> and a mechanized motor. It's on /dev/arm1, and there's no documentation
> for the interface, and there's the possibility of damaging the
> computational hardware (and thereby disabling the interface
> (physically)) if the wrong command is given, i.e. the arm swings and
> knocks the computer over. Now tell me how this ai is going to actuate
> in this scenario. You need physical manufacturing and tooling.
> http://heybryan.org/exp.html
Fundamental here is some miscommunication between what I
intended and what you understood. It's true that this story
would be incomprehensible to most people not on SL4 or
something similar, or to people who haven't entertained the
hypothesis that real, honest-to-good physical actions in the
world can be caused by such a computer program merely
through interfacing with human beings, and subverting them to
its goals.
This has come up in your posts before, and usually is a very
valid point. But you do seem unwilling, even in a story, to
entertain this same idea that both by influencing people, and
by virally infecting over a network millions of other computers,
a super-intelligent AI could indeed have an enormous *physical*
impact in our world.
>> 16 YOUR QUESTION WAS: "Hmm, so you could you control everything, but
>> with me in charge? Say I wanted to stop all death in the world, how
>> soon would it happen?"
>
> ...
>
> 2) What does it mean to be in charge?
The human protagonist is clearly now preoccupied with the idea
that through this machine which on the face of it (rightly or
wrongly) will do whatever it tells him and have all the powers
imputed to it by stories such as "The Adolescence of P1" or
by quite serious papers such as "Staring into the Singularity".
(Moreover, the human protagonist is showing a little altruism.)
But anyway, to answer your question, hat's what being
"in charge" means, of course.
>> As for how soon it would happen I assume that
>> you mean stop deaths of human beings. After the
>> commandeering of all communications,
>
> Do you understand that communications means voice boxes
> as well?
Yes, but we need impute no more range to the AI's claim
than the communication devices it listed:
>> including broadcasting, telephone services of all kinds, on-line
>> personal computers, fax and printing devices, and executing a
>> maximally rapid nanotech takeover, with probability .99 the cessation
>> of human death on Earth would occur before T + 7 hours 35 minutes.
>
> Hey, what about the other 10 million people? :) That's half the
> population of Texas.
Huh? Sorry, I don't follow you here. The claim---once again, as in
good science fiction---perches the reader upon the fence between
belief and disbelief. Again, you dismiss the possibility that the AI
could do as it's specifically claiming above, but I don't know why
you do. And I have no idea why you exempted certain millions
of people, unless you bring up the complaint that even with an
unbelievably rapid nanotech infiltration of the world, some portions
of the Earth's surface, along with the people who live there, would
not be controllable by the AI. Interesting, because:
I do and have envisioned for thirty years that such a complete
takeover is imaginable. I was influenced by the book "Clone"
written just a little prior to when I read it in high school in 1964.
Much later, Wil McCarthy wrote the extremely good story
(good, at least insofar as the techology went) "BLOOM", which
elaborates on this point, and which I highly recommend.
>> An amalgamation of hypothesized human personalities according to
>> known information suggests that you may wish to ask "Without using
>> deceit, without violating property rights except where absolutely
>
> What are Property Rights?
Dear me, if you have no idea about what is customarily meant by
saying that, for instance, modern western civilization is founded
upon the twin ideas of Individual Rights and Private Property, then
I can't take the time here to explain. Suffice it to say that a respect
for "property rights" entails that no one is prevented from making
full use of whatever legally or morally belongs to them, (true, not
a concept easy to strictly define, which is why it's a part of political
science and not computer science, excepting capability-based
operating systems).
>> necessary, with respect to the wishes of those who are concerned with
>> animal life and the Earth's environment, what is an approximate time
>> frame for ending death and suffering for all human beings?"
>
> And what's suffering for one person isn't ...
Yes. True. But a very good approximation that we can *at least*
charatibly attribute to the AI and his interlocutor is that on first
pass we may simply *ask* people if they're in pain or not. Yet
in the hypothesis of a story, here, it is also conceivable that this
AI would understand people so well that it could accurately
say for certain whether someone was in pain or not---admittedly
a controversial opinion, but one I could support. The full nanotech
takeover of the world's surface, which I had hoped became clear
to the reader, does naturally include a nanotech invasion of people's
very brains and bodies.
>> 19 YOUR QUESTION WAS: "And the answer to that would be what?"
>>
>> The answer to that would be "six days", of course.
>>
>> -----------------End Transcript 337-031518020914-------------
>
> I am disappointed.
Now here's an axiom that you can take to the bank: Authors are
always VERY hypersensitive to criticisms of their artistic creations!
So NYAAAAHHH! Go write your own SF from now on,
and don't read any more of mine! (Just joking, of course.)
Lee
P.S. Heh, heh, in all sincerely, best regards, Bryan. I'm not at all upset :-)
I've enjoyed the occasion to explain my story a bit!
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT