From: Scott Yokim (scottyokim@yahoo.com)
Date: Mon Jul 24 2006 - 14:44:29 MDT
In February, Eliezer said:
<quote>Reflective decision theory - a theory of motivationally stable
self-enhancement - is the world's second most important math problem.
The *most* important math problem is how to phrase the motivational
invariant itself. A classical utility function *probably* isn't going
to cut it. My suspicion is that being able to build a reflective
decision system, I would know a great deal more about my options for
motivational invariants, and how to structurally describe those
structurally complex things that humans want - such as "free will" or
"freedom from having one's life path too heavily optimized by outside
sources as opposed to one's own efforts". I am doubtful I can solve the
most important math problem without having solved the second most
important math problem first. Sadly and dangerously, FAI knowledge
*always* lags behind AGI knowledge because AGI is a strictly simpler
problem.</quote>
Not only does one need to (learn how to) specify goal preservation in decision theory, but also to learn how to preserve more than one (conflicting!) goal at a time (do no harm to the human race, fulfill the universe's destiny, etc.).
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT