Re: guaranteeing friendliness

From: Ben Goertzel (
Date: Sat Nov 26 2005 - 15:56:47 MST

Jef wrote:
> > It may be that a theorem-proving-based Friendly mind would not have
> > the subjective experience of "continuity of consciousness" but this is
> > far from demonstrated. My guess is that such a mind *would* have such
> > an experience, although it would also be acutely aware of the
> > limitations of this experience as a summary of what goes on inside its
> > mind, and would also have many other forms of subjective experience
> > that we cannot currently conceive.
> While I don't necessarily disagree with this paragraph, it too seems
> to blur the distinction as earlier. For example, to say that a mind
> "has" the subjective experience rather than a mind "reports" the
> subjective experience. To say that it "has" an experience seems to
> invoke that Cartesian dualism that is so ingrained in human thinking
> but has no functional representation (except as a model of that
> subsystem of human thinking.) It's a powerful illusion. And
> completely real and modelable--the illusion, that is.

The English word "has" is rather ambiguous, which is why e.g. Lojban
decomposes it into 3 separate words denoting various kinds of

What I would say is: there is a subjective experience that is
"uniquely associated" with that mind, in the sense that it is not
associated in the same close way with any other mind. That is what I
mean by "the mind has a subjective experience."

The status of this kind of statement is about the same as when one
says "I see a book on the table." Of course, what's really going on
is that some particles associated with the book are impacting the
retina, etc. etc. -- but it would be rather awkward to say instead,
each time, "My brain generates the phenomenal illusion that there
exists a book in phenomenal observed physical reality that is on the
phenomenal illusion of the table..."

I suppose I am a sort of dualist, in that I think that both
subjectivist and objectivist languages are useful for describing
reality. But I don't believe there is a subjective realm that has its
own special dynamical laws which interact with the dynamical laws of
objective reality -- I have seen no evidence for that kind of dualism.

> My own thinking about advanced minds is that they will be highly aware
> (meaning able to effectively incorporate a wide range of sense data
> into their model) of their world and their internal operation, and
> they will be able to communicate with us in our language, mapping
> misleading and ambiguous human phraseology (when appropriate) onto
> their more precise message, but that they will have no use for the
> illusion of subjective experience since they will have more direct
> access to the more objective information relevant to achieving their
> goals.

I'm not sure, because no reasonably creative, active finite mind will
in practice have access to every relevant piece of knowledge about
itself. Truly complete self-understanding is not going to be possible
for any nontrivial real-world mind, and therefore any mind will need
to use some kinds of approximation models to study and represent
itself. Presumably the approximation models of a very advanced mind
will be less stupid than ours, and how much they'll have in common
with ours (e.g. with approximation models like "free will" and
"stream of awareness") I really don't know...

-- Ben G

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:53 MDT