From: Lee Corbin (lcorbin@rawbw.com)
Date: Wed Jul 16 2008 - 00:13:06 MDT
Wei Dai wrote
>> I wonder if anyone else finds it plausible that two dissimilar
>> SIs can know each other's source code.
Naturally, they may have each other's source code at hand,
and be able to study it at leisure. Doubtless they can even
make some general prognostications. But I'm sure that by
"know" you mean more than that.
>> If we assume that they start out without such knowledge, but each
>> wants the other gain it, what can they do? Can one SI prove to another what
>> its source code is? Or, is there some other argument for why SIs might know
>> each other's source code (beyond "we don't know what SIs might be capable
>> of, so we can't rule it out")?
So perhaps you mean that each is actively altering his own
code in real time. It seems likely that each would be able
to desist from doing that for a little while, and just think
about all the data they have. Let's further assume that they
do make available "their data" to each other, a whole yotta
data, to be sure. Suddenly, then, they now have the same
data.
Krekoski Ross writes
>I have two concerns-- one is one that I've voiced before--- simply,
> can any machine emulate itself perfectly?
It would seem necessary that the answer be "no". Suppose that
a millisecond ago the machine was doing
MOV AL,[SI]
MOV AX,[BX + DI]
INC SI
INC SI
DEC CX
MOV SI,IOBUF
...
and it knows this, so it must load the first MOV instruction,
execute it, store the result somewhere, load then second MOV
instruction, simulate it too, and continue. Clearly, it will never
catch up with itself.
> The concern I have is that
> if a machine can apparently emulate itself, then it has free capacity
> and is not completely emulating itself (i.e. it would have to emulate
> itself emulating itself, which of course eventually runs into problems.)
Yes, problems like the above.
> I suspect that two machines of similar complexity and capacity
> would run into a parallel concern.
Seems clear to me too.
> Secondly, if the first concern turns out not to be a problem, then
> there is the obvious concern of, well, the source code contributes a
> fairly negligible component of overall complexity to the SI in
> question (analogous to our DNA's contribution), and sharing the source
> code would perhaps yield some insight to the behaviour of that
> particular machine....
> We would need to also convey the sum total of all input that
> the machine has received since activation.
Yes, I was calling that its data.
> Perhaps we could consider this to be part of the 'source code' but
> then of course its not really source code in the way that we normally
> construe it to be-- again, this is a distinction I've raised here before--
> are our common metaphors 'memory' 'processing' 'storage' 'code'
> even suitable here?
I think that anyone familiar with computers will recognize the
limitations of some of the metaphors. On the whole, though,
I think that they work rather well.
Anyway, I got the last word in when discussing it with
Eliezer (which is considerably easier to do than it is in
person).
I wrote
Your AIs know each other's *code* but
they do not know each other's present *state*. Therefore
they cannot in confidence complete a model of the other's
behavior, unless "always cooperate with an entity whose
source code I have read and whose source code contains
this statement or its equivalent" is indeed part of the source
code of each. But then, they would no longer have any
option to be affected by other memes, such as the following:
"What will happen if I Defect, [as] the a post *I* just
read on Extropians implies I could?"
Lee
> On Tue, Jul 15, 2008 at 6:14 AM, Wei Dai <weidai@weidai.com> wrote:
>> A couple of months ago Lee Corbin and Eliezer Yudkowsky had a conversation
>> about superrationality on the extropy-chat mailing list [1]. Eliezer took
>> the position that two dissimilar SIs may be capable of superrationality when
>> they know each other's source code. He wrote:
>>
>>> Yes, but in this case a *motive* exists to *deliberately* correlate
>>> your behavior to that of your opponent, if the opponent is one who
>>> will cooperate if your behaviors are highly correlated and defect
>>> otherwise. You might prefer to have the opponent think that your
>>> behaviors are correlated, and then defect yourself; but if your
>>> opponent knows enough about you to know you are thinking that, the
>>> opponent knows whether your behaviors are really correlated or not.
>>>
>>> I'm thinking here about two dissimilar superintelligences that happen
>>> to know each other's source code.
>>
>> Putting aside the issue of superrationality for now, I wonder if anyone else
>> finds it plausible that two dissimilar SIs can know each other's source
>> code. If we assume that they start out without such knowledge, but each
>> wants the other gain it, what can they do? Can one SI prove to another what
>> its source code is? Or, is there some other argument for why SIs might know
>> each other's source code (beyond "we don't know what SIs might be capable
>> of, so we can't rule it out")?
>>
>> [1] http://lists.extropy.org/pipermail/extropy-chat/2008-May/043362.html
>>
>
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT