Re: Investing in FAI research: now vs. later

From: Matt Mahoney (
Date: Wed Feb 20 2008 - 20:59:11 MST

--- Daniel Burfoot <> wrote:

> The scenario I'm most afraid of is not a hard take-off leading to unfriendly
> AGI, but a pseudo-AI falling into the hands of evil men.

Bad science fiction. AI will evolve as a collective intelligence distributed
over the internet, not in some lab. It will be in everybody's hands.

The scenario I'm most afraid of is our own stupidity. CEV is a good idea, but
it won't be implemented because people want their immediate wishes granted
right now, not their extrapolated wishes.

> Furthermore, an intelligent system need not be fully general in order to
> give a small group of people the ability to dominate/enslave humanity. Think
> 1984, except Big Brother is equipped with a sophisticated computer vision
> and speech recognition system, so that the Ministry of Truth instantly knows
> if you've spoken anything ungrammatical (that is, your sentence was invalid
> under the grammar of NewSpeak). The Microsoft Office grammar checker becomes
> rather more sinister in this light.

Orwell did not predict that we would want surveillance. Nor is it necessary
to enslave humans by force. The pleasure of a thousand permanent orgasms is
just a matter of changing a few lines of code in your brain.

-- Matt Mahoney,

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT