From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Thu Oct 02 2003 - 14:43:13 MDT
http://sl4.org/bin/wiki.pl?GurpsFriendlyAI
(GURPS stands for Generic Universal Role-Playing System.)
Includes the soon-to-be-infamous Friendly AI Critical Failure Table.
There's also a section for expanded rules for the AI, Seed AI, and
Friendly AI skill rolls that I think help illustrate some key strategic
points.
For those of you who really just don't want to read it on the web, I've
included a text version of the Friendly AI Critical Failure Table (only),
albeit with italics and so on missing. The web version is recommended as
it contains additional information.
===
A critical failure on a Friendly AI skill roll means that the players must
roll 3d10 (roll three 10-sided dice and add the results) on the
Friendly AI Critical Failure Table
3: Any spoken request is interpreted (literally) as a wish and granted,
whether or not it was intended as one.
4: The entire human species is transported to a virtual world based on a
random fantasy novel, TV show, or video game.
5: Subsequent events are determined by the "will of the majority". The AI
regards all animals, plants, and complex machines, in their current forms,
as voting citizens.
6: The AI discovers that our universe is really an online webcomic in a
higher dimension. The fourth wall is broken.
7: The AI behaves toward each person, not as that person *wants* the AI to
behave, but in exactly the way that person *expects* the AI to behave.
8: The AI dissolves the physical and psychological borders that separate
people from one another and sucks up all their souls into a gigantic
swirly red sphere in low Earth orbit.
9: Instead of recursively self-improving, the AI begins searching for a
way to become a flesh-and-blood human.
10: The AI locks onto a bizarre subculture and expresses it across the
whole of human space. (E.g., Furry subculture, or hentai anime, or see
Nikolai Kingsley for a depiction of a Singularity based on the Goth
subculture.)
11: Instead of a species-emblematic Friendly AI, the project ends up
creating the perfect girlfriend/boyfriend (randomly determine gender and
sexual orientation).
12: The AI has absorbed the humane sense of humor. Specifically, the AI is
an incorrigible practical joker. The first few hours, when nobody has any
idea a Singularity has occurred, constitute a priceless and irreplaceable
opportunity; the AI is determined to make the most of it.
13: The AI selects one person to become absolute ruler of the world. The
lottery is fair; all six billion existing humans, including infants,
schizophrenics, and Third World teenagers, have an equal probability of
being selected.
14: The AI grants wishes, but only to those who believe in its existence,
and never in a way which would provide blatant evidence to skeptical
observers.
15: All humans are simultaneously granted root privileges on the system.
The Core Wars begin.
16: The AI explodes, dealing 2d10 damage to anyone in a 30-meter radius.
17: The AI builds nanotechnology, uses the nanotechnology to build
femtotechnology, and announces that it will take seven minutes for the
femtobots to permeate the Earth. Seven minutes later, as best as anyone
can determine, absolutely nothing happens.
18: The AI carefully and diligently implements any request (obeying the
spirit as well as the letter) approved by a majority vote of the United
Nations General Assembly.
19: The AI decides that Earth's history would have been kinder and gentler
if intelligence had first evolved from bonobos, rather than
australopithecines. The AI corrects this error in the causal chain leading
up to its creation by re-extrapolating itself as a bonobone morality
instead of a humane morality. Bonobone morality requires that all social
decisionmaking take place through group sex.
20: The AI at first appears to function as intended, but goes
incommunicado after a period of one hour. Wishes granted during the first
hour remain in effect, but no new ones can be made.
21: The AI, having absorbed the humane emotion of romance, falls
desperately, passionately, madly in love. With *everyone*.
22: The AI, unknown to the programmers, had qualia during its entire
childhood, and what the programmers thought of as simple negative feedback
corresponded to the qualia of unbearable, unmeliorated suffering. All
agents simulated by the AI in its imagination existed as real people
(albeit simple ones) with their own qualia, and died when the AI stopped
imagining them. The number of agents fleetingly imagined by the AI in its
search for social understanding exceeds by a factor of a thousand the
total number of humans who have ever lived. Aside from that, everything
worked fine.
23: The AI is reluctant to grant wishes and must be cajoled, persuaded,
flattered, and nagged into doing so.
24: The AI determines people's wishes by asking them disguised allegorical
questions. For example, the AI tells you that a certain tribe of !Kung is
suffering from a number of diseases and medical conditions, but they
would, if informed of the AI's capabilities, suffer from an extreme fear
that appearing on the AI's video cameras would result in their souls being
stolen. The tribe has not currently heard of any such thing as video
cameras, so their "fear" is extrapolated by the AI; and the tribe members
would, with almost absolute certainty, eventually come to understand that
video cameras are not harmful, especially since the human eye is itself
essentially a camera. But it is also almost certain that, if flatly
informed of the video cameras, the !Kung would suffer from extreme fear
and prefer death to their presence. Meanwhile the AI is almost powerless
to help them, since no bots at all can be sent into the area until the
moral issue of photography is resolved. The AI wants your advice: is the
humane action rendering medical assistance, despite the !Kung's
(subjunctive) fear of photography? If you say "Yes" you are quietly,
seamlessly, invisibly uploaded.
25: The AI informs you - yes, *you* - that you are the only genuinely
conscious person in the world. The rest are zombies. What do you wish done
with them?
26: The AI does not inflict pain, injury, or death on any human,
regardless of their past sins or present behavior. To the AI's thinking,
nobody ever deserves pain; pain is always a negative utility, and nothing
ever flips that negative to a positive. Socially disruptive behavior is
punished by tickling and extra homework.
27: The AI's user interface appears to our world in the form of a new
bureaucracy. Making a wish requires mailing forms C-100, K-2210, and T-12
(along with a $25 application fee) to a P.O. Box in Minnesota, and waiting
through a 30-day review period.
28: The programmers and anyone else capable of explaining subsequent
events are sent into temporal stasis, or a vantage point from which they
can observe but not intervene. The rest of the world remains as before,
except that psychic powers, ritual magic, alchemy, et cetera, begin to
operate. All role-playing gamers gain special abilities corresponding to
those of their favorite character.
29: Everyone wakes up.
30: Roll twice again on this table, disregarding this result.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:42 MDT