From: Robin Lee Powell (rlpowell@digitalkingdom.org)
Date: Sat Oct 24 2009 - 14:31:48 MDT
On Sat, Oct 24, 2009 at 12:35:29PM -0700, Tim Freeman wrote:
> Here's an example that bothers me: Nobody has lived 1000 years.
> Maybe the human mind has bugs in that untested scenario. One
> possible bug is that all 1000 year olds are suicidal. I'm
> concerned that the Extrapolation step would figure out what people
> would want if they "had grown up farther together", where
> "farther" means 1000 years, and then correctly infer that everyone
> would want to die in that situation. The AI gives them what they
> would want if they had grown up farther together by killing them
> all now. I'd prefer that it let the situation evolve naturally --
> that way maybe people would kill themselves at age 900 but they'd
> still get a decent life for a while.
I would hope that the extrapolation would include extrapolating the
actions of the AI; like saying, "Hey, there's a bug that's going to
make you suicidal in a few years; you want I shoud fix that?".
> Here's another example that bothers me: Mpebna in Somalia is
> starving. If Mpebna weren't starving, and had grown up in a more
> gentle environment, he would like, among other things, to have a
> virtual reality system that allowed him to communicate visually
> with his friends without the trouble of travelling to visit them.
> The FAI comes into power, and poof! Mpebna is presented with a
> fancy VR system. Mpebna doesn't know WTF it is, Mpebna is still
> starving, and now Mpebna hates the people who deployed the FAI,
> since they could have fed him and they chose not to. How exactly
> did the people who deployed the FAI benefit from getting into
> conflict with Mpebna here? The alternative was to give him food
> and wait patiently for him to want something else.
Wait, what? That's a total failure to enact the result; that's
sub-goal stomp of the worst kind. Doing that guarantees that Mpebna
will never get to the Nice Place To Live that CEV envisioned, so
it's a stupid action, contrary to the point of CEV.
The point isn't to extrapolate what people would want and then just
give the end result to them; the point is to extrapolate what people
would want and the create a world that leads naturally to, or at
least easily allows, those things to occur. That is: to find out
what humanity might really want some day as A Nice Place To Live,
and then make sure that that's possible. Not even to force it,
necessarily; just make sure it's possible.
IOW, the end result of CEV isn't "poof, here's a VR system; you'd
eventually want that if you could", it's "here's a box that produces
infinite, and some educational material; I'll come back in a year
and we'll talk about computers", because that's how you get Mpebna
started down the road towards being in a Nice Place To Live.
Very loosely; I'm not a super-intelligent FAI, and I haven't
seriously analyzed that particular example even with my weak-ass
brain; do not pick holes in that particular example please, it's not
the point. Maybe CEV leads to Mpebna suddenly understanding vast
amounts of information in a microsecond, and finding himself
immortal and massively wise; I dunno. But certainly "here's a VR
suit, sorry your starving, bye" would mean CEV was an abject
failure.
The point of extrapolating is to stop CEV from doing things to make
people happy now that would prevent them from getting the best
outcome in the future. If it turns out, for example, that future
humans will have a shared morality of tolerance and mercy; if CEV
simply did whatever current humans want, then it might (for example)
find and utterly destroy all the remaining Nazi officers hiding out
in various places, an action that future humans would predictably
abhor, and that cannot be un-done. A crappy example, admittedly,
but the point is just that without extrapolation we can't avoid
permanent affects we might regret later.
-Robin
-- They say: "The first AIs will be built by the military as weapons." And I'm thinking: "Does it even occur to you to try for something other than the default outcome?" See http://shrunklink.com/cdiz http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT