From: Perry E. Metzger (perry@piermont.com)
Date: Fri Jan 02 2004 - 17:30:59 MST
"Eliezer S. Yudkowsky" <sentience@pobox.com> writes:
> To sum up, natural selection *as we know it*, which is to say, natural
> selection in any noticeable quantity, is not an automatic consequence
> of physics. It applies to butterflies, but not pebbles,
Because pebbles cannot construct new pebbles that inherit their
characteristics. If one can build a Von Neumann Machine and launch it
out to spread its descendents throughout the universe, you can be damn
sure that natural selection applies to it.
> Correspondingly, we can expect that any SI we deal with will exclude
> the set of SIs that immediately shut themselves down, and that
> whichever SI we see will be the result of an optimization process that
> was capable of self-optimization and preferred that choice. But this
> does not imply that any SI we deal with will attach a huge intrinsic
> utility to its own survival.
Then the other SIs out there somewhere that arrive eventually and
which *do* attach a huge utility to their own survival might end up
having an advantage over them. We do not control the set of all actors
that are participating in the experiment. (Indeed, we barely control
any of them.)
> It follows that we have no reason to expect any SI we deal with to
> attach a huge intrinsic utility to its own survival.
Again, there will be (eventually) entire ecologies of new life that
arise outside of the area of our physical control that we'll have to
confront. I do not believe that your argument applies to all of them
even if it were to apply to a anything we ourselves build.
> Similarly, for any optimization process that can configure matter in
> ways that it reckons will create instrumental utility, or fulfill
> intrinsic utility, or avoid expected negative utility, with respect to
> any possible aspect of its goal system, we should expect that
> optimization process to optimize all available matter, since that
> action will be perceived as more desirable than the alternative,
> assuming the entity implements some kind of expected utility equation
> for ordering preferences over choices.
>
> For an extremely large class of SIs, they will *all* choose to absorb
> all nearby matter. So there is no reason to suppose that they would
> need a particular desire to reproduce.
It is difficult to control things at long distances -- the speed of
light gets in the way. That provides a powerful incentive to send out
other intelligences. In addition, in a large enough universe, we can
assume that someone, somewhere, will decide to build Von Neumann
machines eventually, and send them off to explore in all directions.
It is also unclear that we'll deal with only one SI in our local
neighborhood, or that all people building such things will have the
same ideas about how to build them, or that they'll get built by us at
all, etc.
> And finally, there is no reason to suppose that the process whereby
> SIs absorb matter, optimize matter, or in other ways do things with
> matter, would create subregions with (a) large heritable changes in
> properties, that (b) correlate to large differences in the rate at
> which these regions spread or transform other matter, and that (c)
> this process will continue over the thousands or millions of
> generations that would be required for the natural selection dynamic
> to produce optimized functional complexity.
Even if we had any control over what shows up around here, we don't
have particularly good control over the entire reachable portion of
the universe.
> Almost any utility function I have ever heard proposed
> will choose to spread across the cosmos and transform matter into
> either (1) *maximally high-fidelity copies* of the optimization
> control structure or (2) configurations that fulfill intrinsic
> utilities.
In the only experiment we've seen done on this -- the earth -- we
found neither condition to have been satisfied. I'm far from convinced
the future will necessarily work the way you propose.
> In short, under scenarios of the type I have seen discussed so far,
> replicator dynamics do not apply to SIs.
So you keep asserting....
Perry
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:43 MDT