**From:** Stuart Armstrong (*dragondreaming@googlemail.com*)

**Date:** Wed Jul 16 2008 - 08:03:40 MDT

**Next message:**Nick Tarleton: "Re: [sl4] prove your source code"**Previous message:**Stuart Armstrong: "Re: [sl4] Re: Property rights"**In reply to:**Lee Corbin: "Re: [sl4] prove your source code"**Next in thread:**Tim Freeman: "Proof by construction, again (was Re: [sl4] prove your source code)"**Reply:**Tim Freeman: "Proof by construction, again (was Re: [sl4] prove your source code)"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Working from the source code, and the current state (or a recent

state), up to the result "this entity would not defect in most

reasonably expected circumstance" seems to be a highly non-trivial

procedure.

Unless the SI's are for some reason running at a tiny fraction of

capacity, they would not be able to do the simulations needed to check

such a thing; and the quantity of simulations needed increases if you

need to be certain for a long time, or even in usual circumstances.

The "reasonably expected circumstances" is another problem.

Unless, of course, there are mathematical shortcuts, theorems that

could obliviate the need to match every possible input with the

outcome "defects-doesn't defect". Some source codes could allow an

outside SI to figure out, with great confidence, whether the SI would

defect or not. Call such SI's predictable. It may become an advantage

to be a predictable SI, even if the source code is otherwise inferior.

*> Actually, the difficulty I had in mind was the seeming impossibility of
*

*> *proving* one's source code to another. Sure, one SI can just send her
*

*> source code to another, but how does the other SI know that it's not just
*

*> some source code that she wants him to think she's running?
*

In principle, this problem seems insolveable; there is nothing to stop

an SI having a source code that says: "behave like this for ten

thousand years, then switch to this other, secret source code".

In practice, there might be hope. If one SI had the details of how the

other was historically constructed; if she has access to the full

memory of the other, the physical makeup, follows her subroutines, and

is convinced that the source code is robust against being overthrown

by a secret command of the type above, then trust may be possible.

Again, having a mathematically predictable source code may be an

advantage here.

*> It seems that if it *were* possible to prove one's source code to another,
*

*> interesting things besides superrationality, like for example self-enforcing
*

*> contracts, would be possible. (One simply modifies one's source code to
*

*> guarantee attempts to fulfill the contract terms and then prove the new
*

*> source code to the counterparties.)
*

In principle, impossible, in practice possible (and then, finally,

possible in principle). The reason is that unless they are willing to

put enforcing the contract as their only goal, then the same problem

of matching possible circumstances to behaviour becomes apparent. You

don't know what inputs would cause the other to break the contract;

therefore you can never be certain they won't appear during the

carrying out of the contract.

However, in practice it should be quite doable. Given access to the

other's code under reasonable circumstances, getting a certain

expectation of trust seems plausible.

Then once the "reasonable circumstances", "expectation of trust" and

the potential error are expressed in mathematical form, they could be

incoporated into the contract, and then get a rigourously self

enforcing contract.

Stuart

**Next message:**Nick Tarleton: "Re: [sl4] prove your source code"**Previous message:**Stuart Armstrong: "Re: [sl4] Re: Property rights"**In reply to:**Lee Corbin: "Re: [sl4] prove your source code"**Next in thread:**Tim Freeman: "Proof by construction, again (was Re: [sl4] prove your source code)"**Reply:**Tim Freeman: "Proof by construction, again (was Re: [sl4] prove your source code)"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*