From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Jan 04 2004 - 22:03:15 MST
[Forwarded with minor editing from Extropy list.]
Robin Hanson wrote:
>
> Let's see, if there are lots of "SIs" that pop up from ancestral
> civilizations, we might expect variation and selection among them. You
> seem to be arguing that there won't be enough of them varying enough
> over time for this to happen much, at least within the posited class of
> SIs that are maximally capable and quickly grab all the resources they
> can, until they run into a (by assumption equally capable) neighbor,
> at which point they make peace with that neighbor. If so, the
> distribution of what happens at various places in the future would be
> largely determined by the distribution of preferences that SIs begin
> with.
Yup.
> It seems to me that your key assumption is one of very cheap defense -
> once one SI has grabbed some resources you seem to posit that there is
> little point in some other SI, or even a large coalition of them,
> trying to take it from him.
I agree that this is a key assumption. However, the assumption can fail
and still bar natural selection, if there is little variation in
preferences or little variation in resource-grabbing capacity or little
correlation between the two. Since I suspect that intelligence would use
up almost all the potential variation before what we ordinarily think of
as heritable capacities had the chance to operate, natural selection would
not automatically follow even if there were frequent combats.
> Given this, I suppose the rest of your
> scenario might plausibly follow, but I'm not sure why you believe this
> assumption.
I tend to suspect that between two similar intelligent agents, combat will
be too uncertain to be worthwhile, will consume fixed resources, and will
produce negative externalities relative to surrounding agents. Let us
assume that loss aversion (not just in the modern human psychological
sense of aversion to losses as such, but in the sense of loss aversion
emergent in diminishing marginal utility) does not apply, so that a 50/50
chance of winning - which goes along with the argument of intelligent
optimization using up variation - does not automatically rule out combat.
However, there would still be a fixed cost of combat, probably extremely
high; and if we assume variation in preferences, there would probably be
negative externalities to any nearby SIs, who would have a motive to
threaten punishment for combat. Negotiations among SIs are, I think, out
of my reach to comprehend - although I do have some specific reasons to be
confused - but I still suspect that they would negotiate. The point about
large coalitions devouring single cells is interesting (although my
current thoughts about SI negotations suggest that *the choice to form a
predatory coalition* might be viewed as tantamount to starting a war). If
we do have coalitions eating smaller cells, then we have a filterish
selection pressure that rules out all unwillingness or hesitation to form
coalitions - not necessarily natural selection unless there is heritable
variation, which correlates, etc. But beyond that point, it would
essentially amount to gambling, more than combat - will you be part of the
latest coalition, or not? Something like a tontine, perhaps, until there
are only two entities left standing? But where does the non-random
selection come in? What does it correlate to?
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:43 MDT