From: Michael Vassar (michaelvassar@hotmail.com)
Date: Sun Jul 16 2006 - 11:00:47 MDT
>Schmidhuber's OOPS, if I recall correctly, supposedly has no invariants
>at all. If you can prove the new code has "greater expected utility",
>according to the current utility function (even if the new code includes
>changes to the utility function), and taking into account all changes
>that will be adopted by the new code, the new code gets adopted. But
>Schmidhuber is very vague about exactly how this proof takes place.
>My own thinking tends to the idea of a preserved optimization target,
>preserved preferences over outcomes, rather than protected bits in memory.
Why? This would seem to involve
a) an arbitrary departure from optimality, including from flexibility and
potential for compromise should other SIs be encountered. Optimization
target changes may be part of the cooperative strategy in prisoners dillemas
and the like.
b) Possibly a cap on the realization of positive (and negative admittedly)
values which are not encompassed in the inital optimization target but which
could emerge naturally without competing with the initial target.
I'm not sure that concept b) is valid, see arguments against its validity,
and see resemblences between it and the concept of objective morality, but
it does seem to me that a) is definitely a valid concern
OTOH, if preserved optimization targets are much technically simpler to
implement than the OOPS concept, which as you say suffers from vagueness,
that might be a reason to take risks with its long-term viability in cosmic
scale prisoners dillemas etc.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT