From: Michael Wilson (mwdestinystar@yahoo.co.uk)
Date: Sun Sep 19 2004 - 08:02:45 MDT
Though speculation about post-Singularity development trajectories is
usually futile, my recent research has thrown up a serious moral issue
which I believe has important implications for CV. The basic points are
that a normative method of reasoning exists, it and close approximations
thereof are tremendously powerful and any self-improving rational
intelligence (artificial or upload) will eventually converge to this
architecture unless their utility function explicitly prevents this
action.
The problem here is that just about all the human qualities we care about
are actually the result of serious flaws in our cognitive architecture,
and many of these seem to have lossless translations into goal specifications
for a perfectly rational substrate (the basis for Yudkowsky's really
powerful optimisation processes). As humanity self-improves, normative
reasoning (of which appropriately implemented Solomonoff induction is at
the very least a good approximation) is a major attractor; adopting it makes
you as effective as possible at utilising any finite amount of information
and computing power. If there's any sort of competition going on, turning
yourself into a RPOP is the way to win. Unfortunately it also appears to be
the end of most of the stuff we place moral value on. A universe full of
perfect rationalists is a universe where all diversity resides solely in
people's goal systems (which may or may not converge); the qualities of
'insight', 'creativeness', 'willpower' etc will all dissappear as they are
defined against flaws, and goal-system properties such as 'compassion' will
revert to 'did this person have an initial utility function that was
compassionate under renomralisation'? This is on top of the already known
issues with qualia and the illusion of free will; both are results of
specific (adaptive) flaws in human introspective capability which would be
relatively trivial for transhumans to engineer out, but at the cost of
breaking the grounding for the actual (rather than theoretical)
implementation of our moral and legal systems and creating something we can
no longer emphasise with.
The basic question here is 'can we create a Power we can care about?'. A
Yudkowsky RPOP is at least potentially a Power, but it is explicitly
designed to be one we don't care about, as it isn't sentient in a way we'd
assign moral worth to (a decision currently made using our ad-hoc evolved
neural lash-together). What do we need to add to make it volitional, and
what further characteristics would we want to be present in the beings
humanity will become? Are some inherent limitations and flaws actually
necessary in an intelligence order to qualify as something worthwhile?
Less relevantly, is a Nice Place To Live likely to insist that its volitional
sentients have some selection of reasoning flaws in order to create a
diverse and interesting society? This is something of a blow for
rationalists, in that perfect rationality may indeed be hopelessly inhuman,
but isn't there a way to hybridise normative and non-normative reasoning
into a cognitive architecture that is both powerful and morally relevant
(ok, perhaps this is my desire for Cosmic Power coming through :)?
The CV question could be glibly summarised as 'is there a likely incremental
self-improvement path from me to a paperclip optimiser?'. While few people
like paperclips /that/ much, it seems likely that many people would choose
to become perfect rationalists without appreciating what they're losing. If
there is a path to normative reasoning that looks locally good all the way
and reports back that everything is fine when extrapolating, an
implementation of CV that doesn't allow for this may lead us into something
we should consider a disaster.
This issue is ultimately a comprehension gap; a universe of perfect
rationalists might well be rated as valuable inhabitants, but we have no
way of mapping our conception of worthwhile and desirable onto this
basically alien assessment. Along the wild ride that has constituted my
seed AI research to date, my original engineering attitude (focus on
practical stuff that works, everything can be fixed with enough technology)
has had to expand to acknowledge the value of both the abstract (normative
reasoning theory and relevant cosmology) and the humanist (despite all the
hard maths and stuff you have to cover just to avoid disaster, Friendliness
ultimately comes down to a question of what sort of universe we want to
live in).
* Michael Wilson
http://www.sl4.org/bin/wiki.pl?Starglider
___________________________________________________________ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:46 MST