Re: Universalising an AGI's duty of care

From: Russell Wallace (
Date: Tue Jul 19 2005 - 16:10:18 MDT

On 7/19/05, Eliezer S. Yudkowsky <> wrote:
> That would explain the Fermi Paradox.

I don't think so - however few or many inhabited star systems there
may be in our past light cone, it is certain that the vast majority
did not give rise to intelligent life; each of those stars is simply
wasting energy at the rate of thousands to billions of tons per
second; letting resources go to waste like that wouldn't be very
intelligent (unless one follows the 'well, _really_ advanced
civilizations emigrate to hyperspace' style of argument, which as you
observe makes no further predictions even if true).

> But would you really want aliens to
> permit the Holocaust?

No, but that isn't the relevant question. Of course when considering
alien help, we choose an event that is horrific to _us_ and say, well
we'd have liked intervention to stop it. What if the aliens think the
Holocaust is an artistic performance noble in its tragic beauty, and
their dilemma is whether to intervene against the Allies?

(Of course, the whole "noble in its tragic beauty" idea, while
deliberately perverse, is _humanly_ perverse. I imagine if we really
meet aliens, they'll be surprisingly like us in some ways, while
strange in some ways it had never occurred to us there could be
variation on.)

For a less strange idea, suppose the choice is between the Holocaust
and Prime Intellect - which would you choose? Or for another take,
read Iain Banks' Culture short story about the Contact Section survey
of Earth.

> Would you let it occur on some alien world, if you
> could see, and knew, and had the power to stop it?

In the extremely unlikely event that I'm ever called upon to make that
decision, while my emotional reaction would be that I want to stop it,
I'm going to have to think very carefully about all the consequences,
not just the immediate ones, before I decide.

> If third parties stepped
> in to help those who wished help, would you fight to stop them?

No. There's a difference between "I am not confident enough to
intervene" and "I am confident enough to stop other people

> If there is
> any force that interdicts our world, preventing even others from helping us as
> we would wish to be helped, then I cannot consider them as friends.

If it turns out that is the case, I'll reserve judgement until I'm in
possession of more of the facts.

> It goes
> back to the disturbing question of aliens with alien motives successfully
> constructing FAI, a useless hypothesis that explains anything and everything
> and makes no further predictions even if it's true.

Yes, if you follow the logic of the interdiction hypothesis far
enough, it ends up being equivalent to the simulation hypothesis; my
reaction to which is to say "oh, cool" and go about my business as

- Russell

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:51 MDT