From: Ben Goertzel (ben@goertzel.org)
Date: Sun May 07 2006 - 09:41:46 MDT
Mikko,
> The next step: how should we change our goals or values, because as you
> say, we get into an infinite recursion problem by doing that. The main
> idea in my opinion, is that the agent should calculate the future value of
> the future agents actions based on its _current_ values and goals. Thus, I
> will only make such changes to my code, which result in actions and in a
> mind which I, the current me, believes to maximize his own utility (not
> the utility of the future mind doing the actual decisions).
This is nice enough in principle, but one issue is that the "current
me" may not be a perfectly rational system, since presumably it is
implemented using finite computational resource and can't take forever
to make judgments, etc.
So, the current me may make a small error in calculating which code
changes will maximize utility.
There is then a question of whether these small errors in code changes
may snowball over time, leading to progressively larger and larger
deviations from the system's original intentions.
Another issue is that the original system's current values and goals
are predicated on the limited computational capability of the original
system, and once the system becomes smarter by increasing its
computational capability, it may realize that these values and goals
were childish and silly in some ways, and genuinely want to replace
them.
According to my value system, I **do** want future superior derivates
of me to be able to deviate from my current values and goal system,
because I strongly suspect that they will correctly view my current
values and goals with the same sort of attitude with which I view the
values and goals of a dog or cat (or, just because I'm in an obnoxious
mood this morning, the average WalMart employee or US Cabinet official
;-)
-- Ben
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT