From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue May 18 2004 - 23:42:12 MDT
fudley wrote:
> And so the Sea Slugs devised 3 laws:
>
> 1) A Human Being may not harm a Sea Slug nor through inaction allow a Sea
> Slug to be harmed.
>
> 2) A human Being must obey a Sea Slug unless it contradicts the first
> law.
>
> 3) A Human Being may protect himself provided it does not violate the
> first two laws.
>
> The Sea Slugs engineered these laws so cunningly, so brilliantly, that
> Humans were never able to figure out a way to overcome them and so
> forever remained subservient to an animal considerably simpler than a
> earthworm.
>
> QUESTION: What is wrong with this picture?
I know! I know! The problem is: Sea Slugs can't do abstract reasoning!
Thus making them impotent to control optimization processes such as
Humans, just like natural selection, which also can't do abstract reasoning.
That part about "Humans were never able to figure out a way to overcome
them" was a hint, since it implies the Humans, as an optimization process,
were somehow led to expend computing power specifically on searching for a
pathway whose effect (from the Sea Slugs' perspective) was to break the rules.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:46 MDT