From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Mar 21 2001 - 13:10:03 MST
Gordon Worley wrote:
>
> What about an SI that just makes everyone think that it is Friendly?
That's the problem with the idea of Friendly SIs leaving the Sysop.
Either the Sysop decides that it's mathematically possible to be totally
confident in the Friendliness of those who go, or even SIs still work
through the Sysop API. Either outcome is acceptable.
> Such a situation would be dangerous because an un Friendly SI could
> slip thorough and do something horrible, because all that there would
> be to stop ver are Friendly SIs who wouldn't hurt a fly in VR. It is
> going to take un Friendly SIs to stop un Friendly SIs who manage to
> escape the control of a Sysop.
This seems to me like a totally wacky idea. Friendly SIs should be just
as good at chess, or battle tactics, as unFriendly ones. A Friendly SI
has no evolutionary history of going to war for the wrong reasons; ve has
no need to practice nonviolence when battle strategy is required. I'd
expect a Friendly SI and an unFriendly SI to look equally inedible to an
external predator.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT