From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Mar 01 2002 - 09:41:40 MST
Ben, if you want to rephrase it, it's fine. Obviously I have no authority
whatsoever to put the oath in canonical form. But I'd like to see some
evidence that you're taking this seriously. If you like, I'm not asking as
Eliezer Yudkowsky, Singularity specialist; I'm just asking as an ordinary
human being, one of many. It is possible that your future actions may kill
me or worse. I'm asking for a promise from you that you'll do your best to
avoid that. It is unfortunately true that humanity does not presently have
social structures which could handle this problem collectively and that the
problem must be handled by someone anyway, but that doesn't diminish in the
slightest the staggering chutzpah of doing it. Humanity is fully entitled
to demand certain promises from me, and with the declaration you just made,
we are entitled to demand those promises from you as well. I didn't ask you
to swear that you would put 100% of your personal efforts into this. I just
asked you to swear that, insofar as you did put in any efforts, you would
put in those efforts on behalf of humanity and not any other consideration.
There is a difference between wanting to spend some time with your children,
and making a decision that permanently destroys a part of humanity's
potential because you made the decision on the basis of how it affected your
children and not how it affected the whole of humanity, should the issue
somehow arise.
And before anyone in the audience even asks, these issues arise
*immediately* from the *declaration of intent* to do certain things,
regardless of whether you think Ben or I or whoever is probabilistically
likely to succeed in doing them. Maybe I believe in Ben's AI design and
maybe I don't, but since I don't know the whole design, and I know that Ben
isn't the only person on the team, the *conservative* assumption from the
let's-protect-the-Singularity standpoint is to take Novamente's announced
intent seriously.
Incidentally, oath or affirmation is a standard part of the procedure for
swearing in participants in life-critical social processes - judges, juries,
witnesses, political representatives. It's not a "Dungeons and Dragons"
thing.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT