**From:** Lee Corbin (*lcorbin@rawbw.com*)

**Date:** Wed Jul 16 2008 - 00:13:06 MDT

**Next message:**Krekoski Ross: "Re: [sl4] prove your source code"**Previous message:**steven0461: "Re: [sl4] Call for information: Moore's Law"**In reply to:**Krekoski Ross: "Re: [sl4] prove your source code"**Next in thread:**Krekoski Ross: "Re: [sl4] prove your source code"**Reply:**Krekoski Ross: "Re: [sl4] prove your source code"**Reply:**Stuart Armstrong: "Re: [sl4] prove your source code"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Wei Dai wrote

*>> I wonder if anyone else finds it plausible that two dissimilar
*

*>> SIs can know each other's source code.
*

Naturally, they may have each other's source code at hand,

and be able to study it at leisure. Doubtless they can even

make some general prognostications. But I'm sure that by

"know" you mean more than that.

*>> If we assume that they start out without such knowledge, but each
*

*>> wants the other gain it, what can they do? Can one SI prove to another what
*

*>> its source code is? Or, is there some other argument for why SIs might know
*

*>> each other's source code (beyond "we don't know what SIs might be capable
*

*>> of, so we can't rule it out")?
*

So perhaps you mean that each is actively altering his own

code in real time. It seems likely that each would be able

to desist from doing that for a little while, and just think

about all the data they have. Let's further assume that they

do make available "their data" to each other, a whole yotta

data, to be sure. Suddenly, then, they now have the same

data.

Krekoski Ross writes

*>I have two concerns-- one is one that I've voiced before--- simply,
*

*> can any machine emulate itself perfectly?
*

It would seem necessary that the answer be "no". Suppose that

a millisecond ago the machine was doing

MOV AL,[SI]

MOV AX,[BX + DI]

INC SI

INC SI

DEC CX

MOV SI,IOBUF

...

and it knows this, so it must load the first MOV instruction,

execute it, store the result somewhere, load then second MOV

instruction, simulate it too, and continue. Clearly, it will never

catch up with itself.

*> The concern I have is that
*

*> if a machine can apparently emulate itself, then it has free capacity
*

*> and is not completely emulating itself (i.e. it would have to emulate
*

*> itself emulating itself, which of course eventually runs into problems.)
*

Yes, problems like the above.

*> I suspect that two machines of similar complexity and capacity
*

*> would run into a parallel concern.
*

Seems clear to me too.

*> Secondly, if the first concern turns out not to be a problem, then
*

*> there is the obvious concern of, well, the source code contributes a
*

*> fairly negligible component of overall complexity to the SI in
*

*> question (analogous to our DNA's contribution), and sharing the source
*

*> code would perhaps yield some insight to the behaviour of that
*

*> particular machine....
*

*> We would need to also convey the sum total of all input that
*

*> the machine has received since activation.
*

Yes, I was calling that its data.

*> Perhaps we could consider this to be part of the 'source code' but
*

*> then of course its not really source code in the way that we normally
*

*> construe it to be-- again, this is a distinction I've raised here before--
*

*> are our common metaphors 'memory' 'processing' 'storage' 'code'
*

*> even suitable here?
*

I think that anyone familiar with computers will recognize the

limitations of some of the metaphors. On the whole, though,

I think that they work rather well.

Anyway, I got the last word in when discussing it with

Eliezer (which is considerably easier to do than it is in

person).

I wrote

Your AIs know each other's *code* but

they do not know each other's present *state*. Therefore

they cannot in confidence complete a model of the other's

behavior, unless "always cooperate with an entity whose

source code I have read and whose source code contains

this statement or its equivalent" is indeed part of the source

code of each. But then, they would no longer have any

option to be affected by other memes, such as the following:

"What will happen if I Defect, [as] the a post *I* just

read on Extropians implies I could?"

Lee

*> On Tue, Jul 15, 2008 at 6:14 AM, Wei Dai <weidai@weidai.com> wrote:
*

*>> A couple of months ago Lee Corbin and Eliezer Yudkowsky had a conversation
*

*>> about superrationality on the extropy-chat mailing list [1]. Eliezer took
*

*>> the position that two dissimilar SIs may be capable of superrationality when
*

*>> they know each other's source code. He wrote:
*

*>>
*

*>>> Yes, but in this case a *motive* exists to *deliberately* correlate
*

*>>> your behavior to that of your opponent, if the opponent is one who
*

*>>> will cooperate if your behaviors are highly correlated and defect
*

*>>> otherwise. You might prefer to have the opponent think that your
*

*>>> behaviors are correlated, and then defect yourself; but if your
*

*>>> opponent knows enough about you to know you are thinking that, the
*

*>>> opponent knows whether your behaviors are really correlated or not.
*

*>>>
*

*>>> I'm thinking here about two dissimilar superintelligences that happen
*

*>>> to know each other's source code.
*

*>>
*

*>> Putting aside the issue of superrationality for now, I wonder if anyone else
*

*>> finds it plausible that two dissimilar SIs can know each other's source
*

*>> code. If we assume that they start out without such knowledge, but each
*

*>> wants the other gain it, what can they do? Can one SI prove to another what
*

*>> its source code is? Or, is there some other argument for why SIs might know
*

*>> each other's source code (beyond "we don't know what SIs might be capable
*

*>> of, so we can't rule it out")?
*

*>>
*

*>> [1] http://lists.extropy.org/pipermail/extropy-chat/2008-May/043362.html
*

*>>
*

*>
*

**Next message:**Krekoski Ross: "Re: [sl4] prove your source code"**Previous message:**steven0461: "Re: [sl4] Call for information: Moore's Law"**In reply to:**Krekoski Ross: "Re: [sl4] prove your source code"**Next in thread:**Krekoski Ross: "Re: [sl4] prove your source code"**Reply:**Krekoski Ross: "Re: [sl4] prove your source code"**Reply:**Stuart Armstrong: "Re: [sl4] prove your source code"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*