From: Chris Hibbert (hibbert@mydruthers.com)
Date: Sun Nov 26 2006 - 21:57:17 MST
Joshua Fox wrote:
>>> If multiple near-AGIs emerge, then basic Darwinian arguments
>>> show that the one that reproduces itself the best will have the
>>> most copies; and mutations favoring survival will spread.
Eliezer S. Yudkowsky <sentience@pobox.com> responded
>> See the thread "Darwinian dynamics unlikely to apply to
>> superintelligence" in January 2004:
>> http://www.sl4.org/archive/0401/index.html#7506
and Philip Goetz answered:
> I read it, but I didn't understand it. I could not find any logical
> implications in it. A central paragraph is,
>
> "It follows that we have no reason to expect any SI we deal with to
> attach a huge intrinsic utility to its own survival. Why? Because
> that's an extremely specific outcome within a very large class of
> outcomes where the SI doesn't shut itself down immediately. There is,
> in other words, no Bayesian evidence - no likelihood ratio - that
> says we are probably looking at an SI that attaches a huge intrinsic
> utility to its own survival; both hypotheses produce the same
> prediction for observed behavior."
>
> There doesn't seem to be any reason given for the conclusion. I
> could just as well say, "We have no reason to expect any animal we
> deal with to attach utility to its own survival, because that's an
> extremely specific outcome within a very large class of animals who
> don't commit suicide immediately."
The difference is that we have strong reason to believe that animals are
the result of Darwinian forces. The point about an SI is that it's more
likely to be the result of a design process than eons (or millions of
generations) of natural selection. It is natural selection that leads
us to expect a survival instinct.
Phil later wrote:
> Darwinian evolution is only one subclass of the competitive systems
> that lead to self-preservation and greediness. We can expect this
> outcome when there is competition for limited resources, and
> expansion of the use of resources. It doesn't matter whether the
> competition takes the form of a colony of organisms that increase
> their resource share by reproduction, and adapt by evolution; or a
> few superorganisms that increase their resource share by growing, and
> adapt largely by introspection.
But, (absent other forces you haven't specified yet) the drive for
self-preservation is a predictable end-point of a particular selection
process. Have you ruled out other mechanisms? I don't see where how
your argument indicates that the only way to reach superintelligence
also necessarily produces a drive for self preservation. The necessity
of that drive in creatures produced by Darwinian evolution and selection
is pretty well established.
Chris
--
C. J. Cherryh, "Invader", on why we visit very old buildings:
"A sense of age, of profound truths. Respect for something
hands made, that's stood through storms and wars and time.
It persuades us that things we do may last and matter."
Chris Hibbert
hibbert@mydruthers.com
Blog: http://pancrit.org
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:57 MDT