From: Daniel Burfoot (daniel.burfoot@gmail.com)
Date: Wed Feb 20 2008 - 18:59:54 MST
On Thu, Feb 21, 2008 at 4:07 AM, Peter C. McCluskey <pcm@rahul.net> wrote:
> There appears to be a serious lack of communication between people who
> think we're doomed without FAI and the people who expect a diverse society
> of AIs. It appears that the leading advocates of one outcome can't imagine
> how anyone could believe the other outcome is possible. This appears to be
> a symptom of a serious failure of rationality somewhere.
The scenario I'm most afraid of is not a hard take-off leading to unfriendly
AGI, but a pseudo-AI falling into the hands of evil men.
I fear most people dramatically underestimate the impact of evil on the
world. Evil is a Black Swan: a rare event (the majority of people are not
evil) that has huge impact. It is not unreasonable to say that history is
dominated by evil, in spite of its empirical rarity.
Furthermore, an intelligent system need not be fully general in order to
give a small group of people the ability to dominate/enslave humanity. Think
1984, except Big Brother is equipped with a sophisticated computer vision
and speech recognition system, so that the Ministry of Truth instantly knows
if you've spoken anything ungrammatical (that is, your sentence was invalid
under the grammar of NewSpeak). The Microsoft Office grammar checker becomes
rather more sinister in this light.
I also fear that discussion of this point is strongly hindered by the
difficulty we have in talking about politics.
Dan
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT