From: Tim Freeman (tim@fungible.com)
Date: Sun Jun 08 2008 - 17:40:01 MDT
<f21c22e30806030526j16b0a41fpcf38fd2e93a76223@mail.gmail.com>
<68F5968F-43EB-4310-AF6B-6216173E2420@randallsquared.com>
<f21c22e30806040039r5484d230i4c8d75d6a810c309@mail.gmail.com> <38f493f10806040313o3d2cd7a2v764299b48b4cc797@mail.gmail.com>
--text follows this line--
From: "Stuart Armstrong" <dragondreaming@googlemail.com>
>The oracle AI will only be good (in the form discussed here) if the
>theory of Friendliness it comes up with is simple enough that we can
>go through it and check there are no holes.
>
>By personal prejudice is that no such simple theory exists - mostly
>because the definition of friendliness depends on the level of power
>and intelligence of the AI, on the evolution of external
>circumstances, and is not a simple universal.
That's FUD.
There's no oracle required, so far as I can tell, and the solution
isn't incomprehensibly complex, nor is it radically different
depending on the intelligence of the AI. A specification of a
proposed solution is at:
http://www.fungible.com/respect/paper.html
It's not practical, but the added hair to make a practical
implementation isn't likely make it incomprehensible. It has one
known bug:
http://www.fungible.com/respect/paper.html#novice-philosopher-problem
The bug seems fixable. There might be other bugs, but finding and
fixing them doesn't seem likely to make it incomprehensible either.
This proposal may be incorrect, but it's close enough that the correct
solution isn't going to be enormously more complex.
Basically you want an algorithm that implements "figure out what
people want, and do that". Optimizing for a utility function is
simple, in principle, so the hard part is figuring out a utility
function that describes what people want. In ordinary human
discourse, the concept of "what so-and-so wants" is treated as a
simple thing, so the reasonable assumption is that an implementation
of that concept is reasonably simple as well.
It is much more constructive to proceed on that assumption and attempt
to figure out the details than to assume failure and give up before
starting.
-- Tim Freeman http://www.fungible.com tim@fungible.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT