RE: SIAI has become slightly amusing

From: Ben Goertzel (
Date: Fri Jun 04 2004 - 13:24:21 MDT


> > Similarly, I think, a FAI should weight "free choice of sentient
> > beings" pretty high among the values it tries to optimize.
> And you'll pick the weighting yourself?

The weighting should be chosen according to the Golden Section ;-)

> And it'll last for the next
> billion years?

In my view, our engineering plans are only useful for guiding the launch
from the current realm to the next realm.

I don't think that specific weighting parameters chosen by us, at the
launch of the Singularity, are going to persist unto eternity.

A lot of stuff that we can't now forecast is going to happen.

> Ben, I don't think I've ever seen you try to
> think of a
> single thing that could go wrong with *your own* solutions, whatever
> criticism you apply to mine.

Of course a lot of things could go wrong with a Novamente-based AGI.
I'm basically deferring the careful analysis of what these are until
after we have an infrahuman Novamente AGI to play with. Because I think
this analysis will be conductable in a much more intelligent and useful
way at that point.

> I'll make it a challenge: Can you show me a single bit of
> self-criticism,
> a single extrapolation of error or catastrophe in your own
> plans, in any of
> your online pages? (No sudden updates, that's cheating.)

There is plenty of self-criticism in my online writings, for instance

for the section beginning with

"So what did we do wrong? Frankly, all kinds of things. So many
things. It's basically impossible to do anything interesting without
making a lot of mistakes. ..."

You're not going to find many extrapolations of AI catastrophe in my
writings, because as I keep saying, I think it's too early in the
development of AGI for us to be able to make decent predictions of what
will or won't lead to a catastrophe.

-- Ben G

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT