RE: How hard a Singularity?

From: Eugen Leitl (
Date: Wed Jun 26 2002 - 02:56:44 MDT

On Wed, 26 Jun 2002, Stephen Reed wrote:

> I understand from CFAI that one grounds the concept of Friendliness in
> external referents - that the Seed AI attempts to model with
> increasing fidelity. So the evolving Seed AI becomes more friendly as
> it reads more, experiments more and discovers more about what
> friendliness actually is. For Cyc, friendliness would not be an
> implementation term (e.g. some piece of code that can be replaced),
> but be a rich symbolic representation of something in the real world
> to be sensed directly or indirectly.

Thanks for offering digests/interpretations of CFAI for those of us who're
too busy. Much appreciated.

Due to extreme power difference a SI can't help coupling to the ad hoc
consensus ethics encoded in external referents, warping it over the course
of a few iterations. If it eventually detaches from the referents'
floating database the referents' morality will drift away from the
now-fixed SI morality metric.

Assuming, it doesn't start with a grave misunderstanding outright.

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:39 MDT