Re: guaranteeing friendliness

From: Jef Allbright (
Date: Sat Nov 26 2005 - 13:12:13 MST

On 11/26/05, Ben Goertzel <> wrote:
> > Ben, you're framing the question as if there's an objective basis for
> > subjective "continuity of consciousness."
> I believe there is such an objective basis, even though our current
> science of mind is not deep enough to tell us much about this basis.
> By an objective basis, I mean a reductionist explanation for under
> what conditions a mind will describe itself as having the subjective
> experience of "continuity of consciousness."

Okay, then we're in agreement to this point.

> In the case of a series of mindstates transitioning between M and N,
> it is possible that M will describe itself as having the subjective
> experience of "continuity of consciousness" but N will not.

Okay so far.

> In this case, the kind of "continuous awareness thread" I was talking
> about cannot exist.

Your statement asserts the "existence" of a "continuous awareness
thread" that was being reported on, when it would be more accurate to
say simply that each iteration of the mind reported experiencing a
sense of continuity. This is where I think you again begin to mix the
subjective with the objective.

> It may be that a theorem-proving-based Friendly mind would not have
> the subjective experience of "continuity of consciousness" but this is
> far from demonstrated. My guess is that such a mind *would* have such
> an experience, although it would also be acutely aware of the
> limitations of this experience as a summary of what goes on inside its
> mind, and would also have many other forms of subjective experience
> that we cannot currently conceive.

While I don't necessarily disagree with this paragraph, it too seems
to blur the distinction as earlier. For example, to say that a mind
"has" the subjective experience rather than a mind "reports" the
subjective experience. To say that it "has" an experience seems to
invoke that Cartesian dualism that is so ingrained in human thinking
but has no functional representation (except as a model of that
subsystem of human thinking.) It's a powerful illusion. And
completely real and modelable--the illusion, that is.

My own thinking about advanced minds is that they will be highly aware
(meaning able to effectively incorporate a wide range of sense data
into their model) of their world and their internal operation, and
they will be able to communicate with us in our language, mapping
misleading and ambiguous human phraseology (when appropriate) onto
their more precise message, but that they will have no use for the
illusion of subjective experience since they will have more direct
access to the more objective information relevant to achieving their

- Jef

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:53 MDT