From: Rafal Smigrodzki (rafal@smigrodzki.org)
Date: Thu May 29 2003 - 16:04:33 MDT
Bill Hibbard wrote
<huge snip>
The carrot is very effective with well intentioned
> people, which is most people. It is less effective with
> badly intentioned people.
### Forgive me the big snip-out - I do not necessarily disagree with much of
what you write, but I would like to concentrate on the most pressing issue:
You and the government have a very limited ability to discern the
well-intentioned from the badly-intentioned in the context of FAI, and just
as importantly, from the misguided. It is relatively simple to tell a
bank-robber from an ordinary customer (by presence of a gun and by loud
demands for cash, for example). It is still impossible to tell the
difference between the really well-intentioned FAI researcher (i.e. someone
who intends to do good, and truly knows how to do it, as opposed to just
hoping he does), the misguided (good intentions but not smart or not lucky
enough to get the right ideas), and the evil (bad intentions covered with
politically correct verbiage about national security). This uncertainty is
due to our general lack of knowledge about how to build an FAI - you cannot
rationally regulate a non-existent field of research, because all of the
"experts'" guesses are as good as mine or yours at this point. Rational
regulation (both governmental and spontaneous) is only possible once some
experimental knowledge is available, to guide further research in an
incrementally modified trajectory. So, you are incapable of a priori
specifying rational methods of deciding which AGI group to support, and
which to wipe off the Earth, aside from winnowing out the ones with
expressly stated bad intentions - but such thugs are unlikely to be a danger
anyway, for many reasons. The elaborate chip-embedded safety measures are
not wrong (in fact I am a great fan of total transparency) - but they are
missing the point.
It would be an extremely bad idea to specify a single enforceable clause
restricting AGI research to a specific path before it really gets off the
ground. Your suggestion of using "human happiness" as a reinforcement goal
is a case in point - I would see it as a horrible perversion to be dissolved
by nanotechnic goo and stored as a disembodied non-cogitating happiness
circuit, and yet this is exactly the kind of outcome one might expect from a
system designed around the maximization of human happiness.
To make binding recommendations you need to know much more than you do now,
and the only way of learning is doing.
Therefore, it is actually safer to give peer-reviewed support for lots of
different AGI projects, and to defer specific restrictions (to be imposed by
the most competent persons, AGI researchers themselves, and not by
politicians or bureaucrats without a track record of AGI achievement) until
much later in the game. And, simply don't bother with thugs and Prometheans.
Rafal
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:42 MDT