From: Samantha Atkins (samantha@objectent.com)
Date: Sun May 23 2004 - 04:29:40 MDT
On May 22, 2004, at 1:58 AM, Eliezer Yudkowsky wrote:
> Samantha Atkins wrote:
>> On May 21, 2004, at 8:35 PM, Eliezer Yudkowsky wrote:
>>
>>> Also, most of the threats I listed are subtle corruptions, not
>>> easily detectable crashes. It seems to me that if I am too young to
>>> survive without a backup, I am too young to mess around with
>>> self-modification.
>> What? You want to program a FAI seed without so much as a delete key
>> on your keyboard or a source control system? The trick is keeping
>> some trustworthy means of evaluating the latest changes whether to
>> self or to the FAI-to-be for desirability and
>> backtracking/re-combining accordingly. We aren't going to go
>> spelunking into AI or our own minds without at least blazing a trail.
>
> I was speaking of me *personally*, not an FAI. An FAI is *designed*
> to self-improve; I'm not. And ideally an FAI seed is nonsentient, so
> that there are no issues with death if restored from backup, or child
> abuse if improperly designed the first time through.
Funny, but we seem to have brains complex enough to self-improve
extragentically and to augment ourselves in various ways. We also have
the brains (we think) to build the seed of more complicated minds than
our own. I don't see where we aren't designed to self-improve. The
AI will be designed to do it more easily of course.
I do not see that it is ideal to have the FAI seed be nonsentient or
that this can be strictly guaranteed. I don't see how it can be
expected to understand sentients sufficiently without being or becoming
sentient.
>
> Again, I do not object to the *existence* of a source control system
> for humans. I say only that it should be a last resort and the plan
> should be *not* to rely on it or use it.
>
OK, but I was objecting to lack of such for an FAI as you seem to
believe you can think through the design issues so fully as to not need
to backtrack. Many problems in a complex (even if mundane) system
cannot be solved on paper or in one's head satisfactorily. They must
be solved in the developing system itself. From some of your
statements I am not sure you fully understand this.
- samantha
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:36 MST