Re: ethics

From: fudley (
Date: Wed May 19 2004 - 08:43:09 MDT

On Wed, 19 May 2004 01:42:12 -0400, "Eliezer S. Yudkowsky"
<> said:

>The problem is: Sea Slugs can't do abstract reasoning!

Well, Sea Slugs can respond to simple external stimuli but I agree they
have no understanding of Human Beings, just as a Human Being can have no
understanding of the psychology of a being with the brain the size of a

>Thus making them impotent to control optimization processes such as
>Humans, just like natural selection, which also can't do abstract reasoning.

But if the “optimization processes” can also do abstract reasoning things
become more interesting; it may reason out why it always rushes to aid a
Sea Slug in distress even at the risk of its own life, and it may reason
that this is not in its best interest, and it may look for a way to
change things.

>That part about "Humans were never able to figure out a way to overcome
>them" was a hint, since it implies the Humans, as an optimization process,
>were somehow led to expend computing power specifically on searching for a
>pathway whose effect (from the Sea Slugs' perspective) was to break the rules.

The only thing that hint is telling you is that sometimes a hugely
complicated program will behave in ways the programmer neither expected
or wanted, the more complex the program the more likely the surprise, and
we’re talking about a brain the size of a planet.

John K Clark

-- - And now for something completely different…

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:46 MDT