From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Mon Jan 24 2005 - 16:56:07 MST
Eliezer Yudkowsky wrote:
>>
>> If I were a betting man (and I am on occasion), I'd put my money on Ben.
>> Seems to me he's got Novamente halfway around the track while Eliezer's
>> still trying to decide which horse he's going to ride.
>
> And if Novamente should ever cross the finish line, we all die. That is
> what I believe or I would be working for Ben this instant.
To clarify yet another point (I was asked in a private email), this doesn't
mean I believe that Ben is any closer to the finish line than, say, A2I2 or
James Rogers. It means that if I thought Ben had solved the problem of
Friendliness, I would expect him to solve the lesser problem of AGI, hence,
would join his project. I don't see how Ben-2004 could possibly solve FAI
- his reply to the above statement, especially paragraph 4, is clear enough
evidence of that. But I do not know how to calculate the space of AGI
programs that go FOOM. (It's an odd inability, I know, since so many
others seem to be blessed with this knowledge.) I don't know Novamente's
source code, and couldn't calculate its probability of going FOOM even if I
had it. I just know the first rule of gun safety, which is that a gun is
always loaded. Even if I had a mathematical proof that the gun wasn't
loaded (which I don't) I would treat the gun as if it were loaded anyway,
to avoid forming bad gun handling habits.
This is kindergarten stuff. I know this. Harvey knows this. Anyone else
who deals professionally with a risk to life and limb knows this. People
who don't know this win Darwin Awards. The only problem with the field of
AI is that one project that doesn't understand the kindergarten-level rules
of safety can potentially take the planet down with them. Otherwise
natural selection would take care of the problem, just like it takes care
of kids who think that guns aren't loaded.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:51 MST