RE: Strong-willed AIs [was: continuity of self]

From: Ben Goertzel (
Date: Sun Sep 15 2002 - 19:38:17 MDT

> For example, as I understand AI, I
> don't see how
> you could possibly "force-feed an AI with false knowledge" unless you'd
> already brainwashed it into operating the belief-formation system under
> your direction.

I'm guessing that with sufficiently advanced tech it will also be possible
to "force-feed" *humans* with false memories and knowledge.

So I was not assuming a symbolic or semi-symbolic knowledge structure in my

> It seems to me that a seed AI, even without deliberately obfuscating
> itself in any way that would interfere with Friendly AI, can
> easily create
> so many simultaneous dependencies within the system that no human
> programmer is capable of modifying it without AI assistance. Moreover,
> that no human programmer is capable of ripping apart the AI and putting
> the pieces back together in any kind of working order. In fact, I would
> tend to suspect that this would happen naturally and without any
> deliberate planning being required on the AI's part.

I think it's *possible* this will be the case.

I also think it's possible that reprogramming (brainwashing) the mind of a
near-human-level general AI may be possible for a sufficiently clever team
of experts...

To my mind, it's too early to say...

> And there's no
> reason why an AI *couldn't* plan ahead for the contingency of being
> stolen. Things that enable an AI to resist unfriendly external
> modification are not necessarily things that interfere with Friendship
> programmers seeing inside the AI and *consensually* modifying it.

True. And if the AI is just near-human rather than transhuman, there's no
reason why it couldn't plan ahead for this contingency, but be outsmarted by
humans who see through its planning...

-- Ben G

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:41 MDT