From: Ben Goertzel (ben@goertzel.org)
Date: Fri Jun 04 2004 - 12:38:36 MDT
Eliezer wrote:
> Yes, I understand. But I don't trust people's unvarnished
> choices, to
> accomplish what they wish, even to not kill them outright.
> People aren't
> cautious enough. I called this "Murder by genie bottle" and
> I meant it.
The analogy with raising young children seems to work OK here (not
perfectly).
A child isn't always cautious enough either.
I tend to be a permissive parent, which means that I basically let my
kids do what they want. I don't place restrictions on how much TV they
watch, for instance, even though I don't like TV and I think when they
grow up they might look back and wish they hadn't watched as much TV in
their childhood (not that they watch it constantly or anything...).
However, if one of them is about to do something really dangerous or
stupid, then I stop them, invoking the "When you grow up, you'll look
back and be glad I stopped you!" principle.
Similarly, I think, a FAI should weight "free choice of sentient beings"
pretty high among the values it tries to optimize.
The "collective volition" factor should come into play, in effect, as a
mediating factor, to be invoked only when different beings' free choices
collide with each other too severely.
I note that my preference for free choice over extrapolated volition,
wherever possible, is an ETHICAL value choice, it's not something that
can be argued for or against rationally, at least not in a definitive
way.
However, it might be possible to show that having an FAI weight "free
choice of sentient beings" very highly is somehow pragmatically
impossible, whereas having an FAI weight collective volition very highly
is more plausible. I don't believe that this is true, but, Eliezer, if
you think you have a demonstration of this I'd like to see it.
-- Ben G
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:39 MST