From: Samantha Atkins (samantha@objectent.com)
Date: Sun Dec 24 2000 - 02:07:16 MST
"Eliezer S. Yudkowsky" wrote:
dates your train of thought...
>
> Because the human mind has an architecture which admits of no goal
> hierarchy at all, or even a directional goal network. The human mind has
> a grasp of the relation supergoal-of and subgoal-of; there's no
> evolutionary need for things to be organized any more neatly than that.
> And change propagation proceeds slowly relative to the speed of conscious
> thought.
So how do you know that that lack of particular goal architecture isn't
one of its strengths that self-consciousness and mental flexibility
depends on? How do you know your AI will reach comparable consciousness
and go beyond it with a more fixed hierarchical goal system?
>
> A Friendly AI might be trained to refine and build up its Friendliness
> supergoals; there will be subgoals associated with that project; so in a
> technical sense, subgoals are affecting supergoals. But they aren't
> promoting themselves. They are, to use an invalid social analogy, working
> together for the common good on the Supergoal Project.
>
But if the initial statement itself or rather its embodiment in goal[s]
and subgoals is not adequate for its satisfaction then there is little
choice but to rearrange goals themselves. Reword, recast, reprogram.
It is quite possible that former subgoals become elevated in this
process. The supergoal/subgoal hiearachy is a tool, a way of organizing
the system, it is not the Goal itself.
Some problems may require "cheating" in that the goal system has strange
linkages and loops that involve seeming or real contradictions to the
design you propose.
Your above analogy would blow your system architecture apart. Over and
over again we have found that individual freedom to pursue individual
goals does more for the overall good than any sort of top down
organization.
> It should always be possible to choose a perspective which eliminates the
> subgoals entirely, leaving only the consequences of supergoals.
>
I can't quite express it but this conception seems far too fixed and
fragile for an actual working and even superior mind.
- samantha
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT