Re: ESSAY: Goal Preservation

From: Scott Yokim (
Date: Mon Jul 24 2006 - 14:44:29 MDT

In February, Eliezer said:

<quote>Reflective decision theory - a theory of motivationally stable
 self-enhancement - is the world's second most important math problem.
The *most* important math problem is how to phrase the motivational
 invariant itself. A classical utility function *probably* isn't going
 to cut it. My suspicion is that being able to build a reflective
 decision system, I would know a great deal more about my options for
 motivational invariants, and how to structurally describe those
 structurally complex things that humans want - such as "free will" or
 "freedom from having one's life path too heavily optimized by outside
 sources as opposed to one's own efforts". I am doubtful I can solve the
 most important math problem without having solved the second most
 important math problem first. Sadly and dangerously, FAI knowledge
 *always* lags behind AGI knowledge because AGI is a strictly simpler

Not only does one need to (learn how to) specify goal preservation in decision theory, but also to learn how to preserve more than one (conflicting!) goal at a time (do no harm to the human race, fulfill the universe's destiny, etc.).

Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT