From: Wei Dai (weidai@weidai.com)
Date: Fri Jan 02 2004 - 19:27:18 MST
On Fri, Jan 02, 2004 at 06:07:54PM -0500, Eliezer S. Yudkowsky wrote:
> This last point is particularly important in understanding why replicator
> dynamics are unlikely to apply to SIs. At most, we are likely to see one
> initial filter in which SIs that halt or fence themselves off in tiny
> spheres are removed from the cosmic observables. Almost any utility
> function I have ever heard proposed will choose to spread across the
> cosmos and transform matter into either (1) *maximally high-fidelity
> copies* of the optimization control structure or (2) configurations that
> fulfill intrinsic utilities. If the optimization control structure is
> copied at extremely high fidelity, there are no important heritable
> differences for natural selection to act on.
How do you ensure that the optimization control structures *remain*
high-fidelity copies over billions of years? Each control structure will
have to process environmental data as well as communications with
neighboring control structures and the so called "configurations that
fulfill intrinsic utilities" which I presume may be intelligent beings.
All of these will have local variations. Somehow the SI has to ensure that
processing these variations does not cause heritable differences. This
may or may not be possible, but as I suggested earlier the only way it can
succeed is if the SI imposes a strict limit on the kinds of communications
that can occur between neighboring control structures and between control
structures and the configurations that fulfill intrinsic utilities if they
are intelligent beings to prevent memes from being transmitted.
> If there were heritable
> differences, they are not likely to covary with large differences in
> reproductive fitness, insofar as all the optimization control structures
> will choose equally to transform nearby matter.
I disagree with this. Whatever limitations the SI imposes on the control
structures in order to maximize their long term fidelity will have a
negative effect on their reproductive fitness when in competition with
replicators that do not have such limits, and certainly the removal of
these limits is a heritable difference.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:43 MDT