Re: Deliver Us from Evil...?

From: Samantha Atkins (
Date: Thu Apr 05 2001 - 03:24:53 MDT

"Eliezer S. Yudkowsky" wrote:
> James Higgins wrote:
> >
> > I'm not personally convinced that uploading a human first isn't the correct
> > answer. I'm rather inclined to trust a person more than an AI to map out
> > our destiny.
> The problem with uploading is that, chronologically, it arrives
> considerably after both artificial intelligence and military-grade
> nanotechnology. So unless you use a superintelligent AI as assistant,
> uploading can't plausibly come first. But if you can trust an AI to play
> assistant, then you can trust it to be Transition Guide (i.e., essentially
> the same order of risk in both cases).

Of course there are people interested in and working on the upload
scenario that do not believe you require either military-grade NT or an
sentient AI to
scan brains sufficiently to upload.

> I'm also more inclined to trust an AI more than a person... maybe even
> more than I'd trust myself, since I'm not designed for recursive
> self-improvement. Actually, I should amend that: After I've been around
> an AI and had a chance to chat with ver, then I expect to wind up
> justifiably trusting that AI's seed morality around as much as I'd trust a
> human seed morality, and I can also foresee the possibility of standing in
> the AI's presence and just being overawed by vis seed morality. Either
> way, I also expect to wind up trusting that AI's transcendence protocol to
> preserve morality significantly more than I'd trust a human-based
> transcendence protocol to preserve morality.

I don't see how this follows. If you upload a human who quickly
self-improves ver capabilities and becomes an SI and if
super-intelligence brings with it expanded moral/ethical understanding
then I see no reason this combination is less trustworthy than starting
from scratch and only putting in what you believe should be there in the
beginning. Yes a lot of evolved complicated behavior and conditioning
is not present in the AI. But some of that complicated behavior and
conditioning is also the bed of universal compassion and utter

- samantha

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT