From: Ben Goertzel (ben@goertzel.org)
Date: Sat Feb 18 2006 - 20:42:40 MST
Howdy Marcello,
> But, Rice's theorem < http://en.wikipedia.org/wiki/Rice's_theorem > states
> that there is no way to create a checker for every non-trivial property
> (Friendliness is certainly non-trivial in this sense.)
I don't see Rice's Theorem as directly relevant here, because it
doesn't tell you anything about the quality of probabilistic reasoning
achievable by practical intelligent systems under realistic resource
limitation; it's an uncomputability-in-principle sort of result...
> To put it less formally, we'd be giving our Friendliness module the use
> of a genie which is somewhat unreliable and whose reliability in any
> particular decision is, for all intents and purposes, difficult to check.
True
> Knowledge of the AI proper's decision is not sufficient here. The Friendliness
> module would also need the reasoning behind the decision in order to verify
> it.
You have not demonstrated this statement or even argued for it, you've
just stated it...
> However, if the AI proper has a non-verifiable architecture, this
> knowledge may only exist in a crippled form, or, in extreme cases like
> evolutionary programming, not at all.
The idea that evolutionary programming is somehow "nontraceable" in
its production of results is a misconception. Ev. prog. is a
deterministic algorithm and if the intermediate steps in the evolution
process are logged, then it is perfectly possible to mine these logs
and understand why a given answer was arrived at. In fact, mining and
analyzing such logs (albeit usually in relatively simplistic ways) is
the principle underlying Estimation of Distribution Algorithms, an
important subfield of ev. prog. today.
> The only rescue from this mess is to
> make the "Friendliness module" smarter than the so-called AI itself.
a) you have not demonstrated this or even argued for it, just stated it
b) I don't know how you are defining "smarter than" in this statement...
-- Ben
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT