From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Sat Jun 12 2004 - 19:57:23 MDT
Michael Roy Ames wrote:
> Eliezer wrote:
>>
>> Rather, the FAI views its own decision process as an approximation to
>> what extrapolated humankind would decide.
>
> There is no guarantee that an AI with this type of decision process
> would epitomize what I consider human friendliness to be. You had
> "human friendliness" as a first approximation of Friendliness in your
> former writings, thus the word choice: Friendly. You seem to be
> redefining Friendliness as an approximation of CV. You might as well
> rename it CVAI, because its relationship with human friendliness has
> become tenuous.
>
> You seem to be assuming that CV and friendliness are strongly positively
> correlated, but present no basis/evidence for this assumption. If I
> understand your assumption correctly, do you have any support for it?
> In my experience people's decisions bear little correlation with
> friendliness, and appear to be made from a mix of selfish and altruistic
> motivations. What benefit would we derived from creating an CVAI rather
> than an FAI?
>
> CV would appear to be a useful theory, and if realized as a *process*
> would provide useful data on how humanity might make decisions, but it
> doesn't sound in the least bit friendly. I suggest that if it did turn
> out to approximate "human friendliness" better than a very friendly
> human would, then it would be an accident. I do not expect humanity's
> collective volition to be friendly. Do you expect it to be friendly?
> If so, why?
This question does appear to keep popping up. Roughly, a collective
volition is what I get when:
(a) I step back and ask the meta-question of how I decided an earlier
Eliezer's view of "Friendliness" was "mistaken".
(b) I apply the same meta-question to everyone else on the planet.
Whatever it is that you use, mentally, to consider any alternative to
collective volition, anything that would be of itself friendlier - that's
you, a human, making the decision; so now imagine that we take you and
extrapolate you re-making that decision at a higher level of intelligence,
knew more, thought faster, more the person etc.
The benefit of CV is that (a) we aren't stuck with your decision about
Friendliness forever (b) you don't have to make the decision using
human-level intelligence.
It's easy to see that all those other darned humans can't be trusted, but
what if we can't trust ourselves either? If you can employ an
extrapolation powerful enough to leap out of your own fundamental errors,
you should be able to employ it on all those other darned humans too.
Maybe a better metaphor for collective volition would be that it refers
questions to an extrapolated adult humankind, or to a superposition of the
adult humanities we might become.
To be blunt: Yes, the collective volition is supposed to extrapolate far
enough to wash out the stupid. Otherwise, as I currently imagine it, the
Last Judge says "no" and vetoes. The whole Last Judge business is really
ad-hoc, and I hope to replace it with something better, but any
something-better would require at least that level of assurance.
>> *If the FAI works correctly*, then the existence of an FAI is
>> transparent
>
> Transparent in what sense?
In the sense that you'd look at it and say, "Oh, look, there's the
collective volition of humankind," not, "Oh look, there's an FAI."
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT