From: Ben Goertzel (ben@goertzel.org)
Date: Wed Feb 18 2004 - 12:19:31 MST
RE: Humane-ness
Hi,
****
This problem doesn't seem to be "What is the correct property?", or even "How do we get our AGI to preserve this property?”, but rather "What is required in getting our AGI to converge not on the property we define, but on what we intended when we selected the property?" The point I am trying to convey is that if you can achieve this, then the AGI should converge on any supergoal we define, regardless of the goal's complexity, given time and guidance to attain and exceed human equivalent cognition.
****
OK, so you want to make an AI that will study your mind, brain and culture and figure out what you really want from it... and then become that...
The problem is, what you really want may not be possible or self-consistent. You may want certain properties to persist in the AI through iterated self-modification, but it may not be possible for the AI to cause this to be the case... because these properties may not be of the right type to be persistent through iterated self-modification.... Then the AI has to come back to you and say "Sorry, no can do." And then you're stuck with the problem of finding out how to create an ethically positive AI that can really work ... with the AI as a helper...
Anyway, the scenario you describe is a lot easier than what Eliezer is talking about. He's talking about creating an AI that will be "humane" and maintain this humane-ness throughout iterated self-modification. That's probably impossible. But what you describe -- creating an AI that will embody a set of consistent, meaningfully defined goals PRIOR to massive iterated self-modification -- that is not as hard...
-- Ben G
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:45 MDT