From: Wei Dai (weidai@weidai.com)
Date: Tue Dec 31 2002 - 17:07:57 MST
Eliezer suggests in CFAI that a Friendly AI may decide to divide the solar
system into six billion equal parts and give each person property rights
over a six-billionth share.
Why equal shares, instead of shares proportional to current net worth in
US dollars, contribution to the development of the Singularity,
popularity, or some other measure of moral worth? Clearly the overriding
principle being applied here is moral symmetry. But Eliezer goes on to say
that each new person born after the Singularity would only be guaranteed a
Minimum Living Space, which presumably is much smaller than a
six-billionth share of the solar system. So there's an obvious asymmetry
across time.
An even greater asymmetry is that only existing individuals get a share.
Potential individuals, people who may have been born but were not due to
historical accidents, are completely dispossessed. If an SI really cared
about moral symmetry, should it not be symmetrical across the entire
configuration space of possible intelligences? I imagine it would sample
this space according to a uniform distribution, instantiate the
randomly selected individuals, and give each one an equal share of the
universe. No one would be allowed to reproduce in any other way.
A remaining question is, assuming the colonizable universe is limited in
computational resources, how many individuals should be instantiated? Or
equivalently, how much of the universe should each person be given? The
principle of moral symmetry does not seem to provide definite guidance
here. One possibility is that each individual would be instantiated just
long enough for it to have one conscious experience. This would minimize
the inequality between the lucky ones who are chosen and the unlucky ones
who are not.
More seriously, the Friendly AI would have to divide up the universe
somehow. I really don't see a method to do it that does not seem arbitrary
or absurd in some way. It would be a treat just to see what it actually
decides upon, and why.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:41 MDT