RE: Adaptation brings unFriendliness

From: Michael Vassar (
Date: Mon Nov 13 2006 - 21:47:08 MST

Nick Bostrom's paper "The Future of Human Evolution"
addressed this issue in depth. It's a possible problem.
However, it seems much more likely to me that take-off will be very hard,
and thus that evolutionary selective pressures will be minimal.

>Here's a question I have been wondering about. I'm sure that it has been
>addressed somewhere. If someone can point me to a URL, I'd appreciate it.
>If multiple near-AGIs emerge, then basic Darwinian arguments show that the
>one that reproduces itself the best will have the most copies; and
>mutations favoring survival will spread. (Reproduction here means building
>the next generation of technology, based on the previous system and perhaps
>with its help.)
>Yet clearly mutations that involve destroying and/or using the resources of
>(potential) competitors are often adaptive. Thus, AGIs that are not only
>unFriendly but downright aggressive will emerge.
>Note that this has nothing to do with evolutionary algorithms for AI
>development, nor is this an argument that aggression or any sort of goal is
>necessarily built into intelligence. It is just a straightforward Darwinian
>argument that adaptive systems tend to produce "redness in tooth and claw."
>I suppose that in a hard takeoff, the leading AGI could gain so much
>(possibly Friendly) power to make all questions moot. But otherwise,
>the above suggest that we do have some idea of the direction in which AGIs
>will tend to develop within the space of possible intelligences, and that
>it's not a good one? This would be yet another strong reason to be
>for AGI Friendliness.

Add a Yahoo! contact to Windows Live Messenger for a chance to win a free

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:57 MDT