From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Mar 28 2001 - 17:56:21 MST
Chris Cooper wrote:
>
> Thus, no Friendliness conflict. I still
> don't understand why a Friendly SI will be interested in "bootstrapping"
> humans into the post-Singularity toybox, considering that it would be less
> trouble to just copy vimself a few billion times instead.
"Verself". Ve/vis/ver/verself. Check the glossary under "gender-neutral
pronouns". Not that yours is intrinsically worse, or anything; just that,
if we're going to violate the rules, we may as well violate them in a
standardized way.
Anyway, yes, it would be less trouble for the SI to copy verself, and it
would be even less trouble to do nothing at all, but neither would be
maximally Friendly. Under Friendliness, the future in which your volition
to "play in the post-Singularity toybox" is fulfilled is, intrinsically,
desirable, because it's your volition. Volition-based Friendliness has a
positive (volition fulfillment) as well as negative (violation prevention)
aspect.
> If
> the entire human upload/upgrade scenario is based on the strength of
> Friendliness during the AI- to- Transition guide- to- Sysop evolution, I
> hope that everyone involved does a damn good programming job.
I'm not sure I get this. Even on an individual scale - single people, you
and I, hoping to become transhuman - you're still relying on the
persistence of *something*. It may be your entire nature, rather than
just your altruism, but you're still relying on something - the
persistence of your desire to be transhuman, if nothing else.
Also, remember the task isn't infinite; it's only until a finitely
transhuman AI has a pretty good idea of what we're aiming for with all
this "Friendliness" stuff - and more importantly, how we're doing it - so
that the AI can take it from there, and make the same decisions we would
make, only better.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT