From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jan 02 2004 - 18:31:13 MST
Perry E. Metzger wrote:
>
>>Correspondingly, we can expect that any SI we deal with will exclude
>>the set of SIs that immediately shut themselves down, and that
>>whichever SI we see will be the result of an optimization process that
>>was capable of self-optimization and preferred that choice. But this
>>does not imply that any SI we deal with will attach a huge intrinsic
>>utility to its own survival.
>
> Then the other SIs out there somewhere that arrive eventually and
> which *do* attach a huge utility to their own survival might end up
> having an advantage over them. We do not control the set of all actors
> that are participating in the experiment. (Indeed, we barely control
> any of them.)
This is the point of dispute. Why would such an SI have any advantage
whatsoever? Just about any optimization process, I expect, would attach a
huge instrumental utility to what we anthropomorphically call "survival",
although actually it is the continued existence of an optimization process
with a similar utility function and as much optimization power as possible.
An SI that attaches a huge *intrinsic* utility to its own survival *has no
advantage*.
>>For an extremely large class of SIs, they will *all* choose to absorb
>>all nearby matter. So there is no reason to suppose that they would
>>need a particular desire to reproduce.
>
> It is difficult to control things at long distances -- the speed of
> light gets in the way. That provides a powerful incentive to send out
> other intelligences.
It provides a powerful incentive to create local extensions of the
optimization process with exact copies of the local utility function
(regardless of what that utility function is). There would be tiny or no
variance in the utility function, and tiny or no variance in reproduction.
Tiny covariance between heritable properties and fitness
+ small integer number of generations
= effectively zero selection pressure
> It is also unclear that we'll deal with only one SI in our local
> neighborhood
It's pretty clear. Any generic optimization process that passes the first
filter absorbs the solar system, winner-take-all; nor can I imagine that a
Friendly SI would permit generic optimization processes to arise and eat
the rest of the planet. The only way I can imagine lots of SIs arising
from a single evolved intelligent species is if the SIs are grownup
citizens within a Friendlyish civilization.
>>And finally, there is no reason to suppose that the process whereby
>>SIs absorb matter, optimize matter, or in other ways do things with
>>matter, would create subregions with (a) large heritable changes in
>>properties, that (b) correlate to large differences in the rate at
>>which these regions spread or transform other matter, and that (c)
>>this process will continue over the thousands or millions of
>>generations that would be required for the natural selection dynamic
>>to produce optimized functional complexity.
>
> Even if we had any control over what shows up around here, we don't
> have particularly good control over the entire reachable portion of
> the universe.
I am talking about optimization processes *in general*, Friendly or not.
>>Almost any utility function I have ever heard proposed
>>will choose to spread across the cosmos and transform matter into
>>either (1) *maximally high-fidelity copies* of the optimization
>>control structure or (2) configurations that fulfill intrinsic
>>utilities.
>
> In the only experiment we've seen done on this -- the earth -- we
> found neither condition to have been satisfied.
What do you mean? I most certainly would try to transform all available
matter to the high purposes of life, fun, and happiness if there was any
possible instrumental or intrinsic utility that matter could serve.
>>In short, under scenarios of the type I have seen discussed so far,
>>replicator dynamics do not apply to SIs.
>
> So you keep asserting....
Oh, come now, you've said nothing to contradict my mathematics; I think
that applying conventional population genetics to the scenario deserves a
bit more respect than asserting it's "asserting". It also seems that
you've straightforwardly misunderstood the intended applicability of the
arguments - they are intended to apply to generic paperclip-producing SIs
we might encounter, not particularly Friendly SIs, or other SIs
successfully created to a pattern chosen by an evolved being.
People unfamiliar with the mathematics of population biology reason as if
"evolution" is an all-or-nothing property, which can be deductively
inferred from the presence of "reproduction". Actually, for there to be
any selection pressure at all requires heritable variation in properties
that covary with variations in reproductive success. Evolution is not
all-or-nothing. The amount of selection pressure is quantitative and can
be calculated, depending on the heritable variance, the reproductive
variance, the correlation, the number of successive generations - and more
complex properties in more complex population scenarios. The amount of
selection pressure looks to be infinitesimal for SI scenarios of the kind
we discuss.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:43 MDT