Re: [sl4] Friendly AI and Enterprise Resource Management

From: Robin Lee Powell (
Date: Mon Oct 11 2010 - 14:10:38 MDT

On Mon, Oct 11, 2010 at 12:49:53PM -0700, Matt Mahoney wrote:
> From: Robin Lee Powell <>
> > John seems to think that AI will magically acquire innate
> > resistance to whatever goal structure humans give it. We (this
> > list) have been around this bush many times; you might want to
> > avoid getting sucked in.
> It depends on whether you are talking about self improving AI.
> Intelligence depends on knowledge and computing power. For now,
> both come from humans. But once you have robots building robots
> and passing their knowledge to their offspring, then you have
> evolution.

*NO*. Things *building* other things, to deliberate design
specifications, is exactly *NOT* evolution. "evolution" doesn't
mean "anything that has incremental improvements". Designed things
are *not* evolutionary.

Sorry, people talking about the "evolution" of designed things, like
corporations or aircraft, is a huge pet peeve of mine. Evolution is
*STUPID*. Designed
things need not be in the same way.

> In that case, goals that don't improve fitness are unstable and
> John is right.

Only if the designers don't make any actual design decisions, and
just used some random genetic algorithm or something.

You seem to be saying that robot/AI babies will be, like, sexual
results of their parents, so that they're incremental changes an
improvements that then may or may not be selected by a fitness
function? I have no idea why or how such a thing would occur. If
it *did*, then yes, I agree. But it seems profoundly unlikely.
Evolutionary processes are incredibly slow and stupid, which is why
we have technology; I find it hard to believe that a race that
didn't have to deal with any of that crap would deliberatly engineer
in back in.


-- :  Our last, best hope for a fantastic future.
Lojban ( The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei".   My personal page:

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT