Re: ESSAY: Goal Preservation

From: Scott Yokim (scottyokim@yahoo.com)
Date: Mon Jul 24 2006 - 14:44:29 MDT


In February, Eliezer said:

<quote>Reflective decision theory - a theory of motivationally stable
 self-enhancement - is the world's second most important math problem.
 
 
The *most* important math problem is how to phrase the motivational
 invariant itself. A classical utility function *probably* isn't going
 to cut it. My suspicion is that being able to build a reflective
 decision system, I would know a great deal more about my options for
 motivational invariants, and how to structurally describe those
 structurally complex things that humans want - such as "free will" or
 "freedom from having one's life path too heavily optimized by outside
 sources as opposed to one's own efforts". I am doubtful I can solve the
 most important math problem without having solved the second most
 important math problem first. Sadly and dangerously, FAI knowledge
 *always* lags behind AGI knowledge because AGI is a strictly simpler
 problem.</quote>

Not only does one need to (learn how to) specify goal preservation in decision theory, but also to learn how to preserve more than one (conflicting!) goal at a time (do no harm to the human race, fulfill the universe's destiny, etc.).

 __________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com



This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT