From: Ben Goertzel (ben@goertzel.org)
Date: Wed May 26 2004 - 21:52:04 MDT
Eliezer,
About "enthusiasm and comforting ignorance" ..
-- I'm well aware of my tendency to be overoptimistic, and I make a
strong conscious effort to counteract that. Generally I'm successful at
counteracting it these days; for instance, I'm now usually able to make
realistic, non-over-optimistic timing estimates for software projects.
-- I don't find my ignorance very comforting. However, I am comfortable
accepting that I am ignorant about some things! Such as, for example,
whether it's true that, as you claim, "any self-improving AGI not
carefully engineered according to a rigorous FAI theory will lead to a
bad end for humans." I don't know. I don't believe you really know
either, with any reasonable degree of certainty. This ignorance is not
comforting, just realistic.
> I didn't say my insights were hard to grok, Ben, but neither,
> it seems, are
> they so trivial as to be explained without a week of work. I
> say something
> that I see immediately, and you say no. Past experience
> shows that if you
> and I both have the time to spend a week arguing about the
> subject, there's
> a significant chance I can make my point clear, if my point
> is accessible
> in one inferential step from knowledge we already share.
Eliezer, it is unwarranted nastiness on your part to insinuate that I am
not able to follow more than a single inference step. Gimme a break!
> The
> case of AIXI
> comes to mind; you made a mistake that seemed straightforward
> to me because
> I'd extensively analyzed the problem from multiple
> directions. And no, my
> insight was not too subtle for you to comprehend. But it
> took a week, and
> the clock went on ticking during that time.
Heh. That "week arguing about AIXI" that you mention was about an hour
of my time altogether; as I recall that was a VERY busy week for me, and
I was reading and writing emails on the SL4 list in spare moments at
high speed.
> When I came to Novamente, I didn't succeed in explaining to
> anyone how
> "curiosity" didn't need to be an independent drive because it
> was directly
> emergent from information values in expected utility combined
> with Bayesian
> probability. Maybe you've grown stronger since then.
First of all, you never came to Novamente LLC (which BTW does not have
an office, though office space is shared with partner firms in the US
and Brazil). You came to Webmind Inc., which was a different company
building a different AI system.
Novamente is founded on probability theory, Webmind was not. That is
one among several significant differences between the two systems.
Secondly, when you visited Webmind, you (surprise, surprise!) seem to
have come away with the impression that all of the staff were much
slower and stupider than they actually are.
Regarding the particular point you mention, many of us UNDERSTOOD your
point that you COULD derive curiosity from other, more basic
motivations. However, we didn't agree that this is the BEST way to
implement curiousity in an AGI system. Just because something CAN be
achieved in a certain way doesn't mean that's the BEST way to do it --
where "best" must be interpreted in the context of realistic memory and
processing power constraints.
Novamente is more probabilistically based than Webmind, yet even so we
will implement novelty-seeking as an independent drive, initially,
because this is a lot more efficient than making the system learn this
from a more basic motivation.
> But as far as I can tell, you've never
> understood
> anything of Friendly AI theory except that it involves
> expected utility and
> a central utility function, which in the past you said you
> disagreed with.
Well, I believe I understand what you say in CFAI, I just don't see why
you think that philosophy would work in a real self-modifying software
system...
> I still haven't managed to make you see the point of
> "external reference
> semantics" as described in CFAI, which I consider the Pons
> Asinorum of
> Friendly AI; the first utility system with nontrivial
> function, with the
> intent in CFAI being to describe an elegant way to repair
> programmer errors
> in describing morality. It's not that I haven't managed to
> make you agree,
> Ben, it's that you still haven't seen the *point*, the thing
> the system as
> described is supposed to *do*, and why it's different from
> existing proposals.
Your "external reference semantics", as I recall, is basically the idea
that an AGI system considers its own supergoals to be uncertain
approximations of some unknown ideal supergoals, and tries to improve
its own supergoals. It's kind of a supergoal that says "Make all my
supergoals, including this one, do what they're supposed to do better."
What I don't understand is why you think this idea is so amazingly
profound. Yes, this attitude toward one's supergoals is an element of
what people call "wisdom." But I don't see that this kind of thing
provides any kind of guarantee of Friendliness after iterated
self-modification. Seems to me that an AGI with "external reference
semantics" [TERRIBLE name for the concept, BTW ;-)] can go loony just as
easily as one without.
But if I ever disagree with one of your ideas, your reaction is "Well
that's because you don't understand it." ;-p
BTW, the idea of goals having probabilities, propagating these to sub
and supergoals, etc., was there in Webmind and is there in Novamente.
Goal refinement has been part of my AI design for a long time, and it
was always applied to top-level supergoals as well as to other goals.
> Doesn't excuse every new generation of scientists making the
> same mistakes
> over, and over, and over again. Imagine my chagrin when I
> realized that
> consciousness was going to have an explanation in ordinary, mundane,
> non-mysterious physics, just like the LAST THOUSAND FRICKIN'
> MYSTERIES the
> human species had encountered.
1) I don't call quantum physics "non-mysterious physics"
2) Have you worked out a convincing physics-based explanation of
consciousness? If so, why aren't you sharing it with us? Too busy to
take the time?
-- Ben G
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT