From: Rafal Smigrodzki (rms2g@virginia.edu)
Date: Mon Dec 23 2002 - 11:35:36 MST
Cliff wrote:
>
> To be honest, I have some doubts about the Ben5 scenario myself. I'm
> not at all yet certain that the kind of consciousness we are isn't
> inherently complexity-limited in some ways. A paranoid personality
> allowed to evolve and self-modify would quickly spin into some form of
> self-destruction, I'd imagine...so given infinite self-modifiability,
> wouldn't the smallest flaws in our personalities or mental health
> spiral out of control? Perhaps we could modify our personality in
> such a way to avoid this, but that presumes we know how to handle the
> self-modification tools and know what to change and how, which in turn
> presumes we're "given" those tools and that knowledge by an external
> process such as an FAI. I can't quite picture us bootstrapping it all
> the way given just the "mental room"...
### Very good points. Aside from getting a little help from the Friendly AI
and other grown-ups, there is a way of dealing with the risks from
self-modification: copying at each major redesign, so the new version can
start exploring its capabilities, while the remaining parts of your spawn
look closely for any signs of malfunction. In case of major problems, the
spawn would sever the links to the new version, and make a mental note not
to use the particular modifications again. There is a similar approach in
Greg Egan's "Diaspora", where the infrastructure of the polis charged with
production of new citizens keeps tracking the performance of each design and
adjusts the frequency of its shapers accordingly.
Rafal
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:41 MDT