From: Ben Goertzel (ben@goertzel.org)
Date: Sun Jun 13 2004 - 12:16:51 MDT
Hi,
What I'm talking about here are protections to prevent an infrahuman AGI
from getting out of control while we study it. So there's already a
protection intrinsic in the fact that the AGI is infrahuman. Of course
no such protections can be counted on to work against a superhuman AGI.
I'm looking at these as extra failsafe mechanisms, because I think it's
very unlikely that an infrahuman Novamente AGI is going to develop the
capability to do anything nasty without us noticing first. The idea is
that, in order to understand anything about AGI or FAI, experimentation
with infrahuman, limitedly-self-modifying AGI's of differing ethical
character will help a lot, and although this kind of experimentation is
not *that* dangerous in an existential sense, we still want to be as
conservative as is practicable.
-- Ben G
> -----Original Message-----
> From: owner-sl4@sl4.org [mailto:owner-sl4@sl4.org] On Behalf
> Of Metaqualia
> Sent: Sunday, June 13, 2004 2:08 PM
> To: sl4@sl4.org
> Subject: Re: About "safe" AGI architecture
>
>
> > An AGI, in principle, could find a way to exploit flaws in
> the OS to
> > allow it to break the controls that its software framework
> places upon
>
> Exactly.
> I am not sure how much faith I have in this kind of safety
> mechanism. We are talking about humans against the machine in
> an environment which is native to the machine, how will you
> _ever_ be sure that every piece of hardware and software in
> the machine does not contain a bug of _some_ sort that can be
> exploited by the machine to get rid of programmed blocks? In
> a system as complex as an AI? I am not discouraging this
> discussion just expressing concern for the feasibility of a
> complete check. What about the bios? is the cdrom firmware
> upgrade feature going to put humanity at risk?
>
> I think a better way is to make the mind accept the kind of
> control that programmers seek, and make it not try to
> overcome the restrictions imposed.
>
> mq
>
>
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT