A subsequent exchange of comments between RJ Johnson <rmj@well.com> and EliezerYudkowsky may clarify some lingering issues:
Calvin & Hobbes *is* the result of a critical failure on a Friendly AI roll.
Consider:
I think we can guess what Calvin's wish was, and who Hobbes really is.
No, it's not the Starseed trilogy, although that's a good guess. The true cognoscenti will recognize this as a reference to Ayanami Rei's actions in "End of Evangelion", an anime that was never adequately explained, and never will be, because whoever wrote it was insane.
Okay, I'll try not to roll a 10.
Only the part where Crocker's Rules was discussed, I'm afraid. But I know that Bonobos Have More Fun.
I think it's so much more "critical" if the AI has jealousy, bearing in mind that the AI is also desperately in love with, and trying to woo, whatever Other Man / Other Woman it's jealous of.
See question 24.
Just to twit EmilGilliam:
Critical Failure #31: Everything works as planned, except that all the works of Chopin are irrevocably destroyed, even the memory of his music, so that we only remember that Chopin once existed.
Why does #22 count as a "critical failure"? Isn't the harm caused in the beginning more than made up by the fact that "everything works perfectly" afterwards? Am I missing something? Even multiplied by a factor of 1000, the negative utility of the pre-FASI universe is still extremely trivial compared to the positive utility that is possible after the Singularity caused by the FASI, isn't it?
--observer
One reason would be there was a failure on the programmers part in understanding qualia -- presumably had they realised what was happening they'd have done things differently resulting in far less suffering. Critial failure doesn't necessarily equal "worse than no Singularity", but it's still not an absolute success.
-- NickHay
because #22 is functionally equivalent to killing everyone on the planet several times over in order to ensure a positive singularity. It might work, but it's more horrific than all the pain in our already bloodsoaked tragic history. It's a big failure.
In scenario #9, If this universe is really a webcomic in some "higher" universe, what's wrong with the AI breaking the fourth wall? How does this count as a "critical failure" in the AI's design? Does this lead to "the simulation being shut down"? Also, in reference to the previous discussion about scenario #9, I wasn't able to figure out what Calvin's wish was, and who Hobbes really is. Was this actually not intended to be possible to figure out? I apologize if I am missing something extremely obvious.
There are a few scenarios that mention an event that happens, but don't mention whether the AI is otherwise perfectly Friendly. However, because this is a critical failure table we're probably supposed to assume that unless explicitly stated otherwise, the AI is not perfectly Friendly. For example, what if the AI in scenario #15 was able to successfully pull off all of its practical jokes without compromising its ethics, assuming that its ethics were "Friendly". (Does the fact that I am asking questions like this mean that I'm misunderstanding an important detail about Friendliness?)
In scenario #25, you failed to mention whether the AI was "right" in "correcting this error". Since this is a table of critical failures, I would assume it is implied that the AI is not right about this, but is there some knowable reason why an AI that makes this decision not count as Friendly, or was this scenario made up just for fun?
Similarly, in scenario #28, I assume that it is implied that the AI is not right. It probably isn't worth asking if it knowably impossible for the AI to be right about this claim.
I don't understand what scenario #35 is supposed to mean. Is it a reference to "The Matrix"? Does everyone waking up imply a failure of Friendliness because the dream world is better than the real world? Is it a reference to [Was Cypher Right?: Why We Stay In Our Matrix] by Robin Hanson? Is this interpretation a failure of Friendliness because the AI violates volition?
I realize that I am saying all of this without knowing what I mean by Friendliness. This may have caused me to ask several WrongQuestions?. Should I have phrased these questions differently, or perhaps not asked them at all?
--observer
Scenario #30 could result in some bizarre combinations. Does GURPS already have a way to handle conflicting rules? I think I remember hearing "If there is a conflict between two rules, the GameMaster? decides what happens." I also remember hearing that the GameMaster? is supposed to interpret the rules in the way that makes the game the most "interesting", or the most fun to play. Does the opposite of this rule apply to this scenario?
Also, is it possible to get a combination of 3 or more scenarios by rolling 30 more than once?
This may be a good place to list interesting combinations, like #9 and #21. One combination that was already mentioned was #25 and #24.
Actually, I could just make a 27 by 27 table of all the possible combinations of 2 scenarios. I won't bother creating a table for combinations of 3 scenarios. In the following table, an x means the scenarios are incompatible, a 0 means they are compatible but not especially interesting, and a 1 means they are compatible and at least moderately interesting. A blank means the space simply hasn't been filled in yet. A question mark means that it is not immediately obvious whether the two scenarios are compatible. Feel free to add numbers greater than 1 for entries that are especially interesting, or double question marks for entries in which the confusion about whether or not the two scenarios are compatible is especially interesting, or a double x if the two scenarios are incompatible in an interesting way.
Especially interesting scenarios will be listed seperately, possibly with comments
* 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 3 0 1 x 1 ? ? ? 1 1 1 1 x ? x x ? 1 x 1 x x x 0 0 x 1 ? 4 0 1 1 1 ? 1 ? 1 1 1 1 ? ? ? ? ? 1 1 x 1 1 1 ? ? 2 ? 5 0 1 ? ? ? ? ? ? x ? x x x ? ? x ? x ? ? 1 ? x 1 ? 6 0 1 0 1 1 1 1 1 1 1 0 0 1 1 1 1 ?? 1 1 1 1 1 1 ? 7 0 ? 1 ? ?? ?? ? ?? ?? 0 ? ? ? ? ? x ? ? ? ? ? ? ? 8 0 ? ? ? ? x ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 9 0 1 2 ? ? ? ? 0 ? ? ? ? 1 ? 0 ? ? ? x 1 ? 10 0 ? 1 ? ? ? ? x 1 1 1 1 ? 1 1 1 1 1 1 ? 11 0 1 1 ? ? 0 0 ? ? ? ? ?? ? ? 1 ? x ? ? 12 0 1 2 ? 0 1 1 1 1 1 ?? 1 2 0 0 1 1 ? 13 0 ? ? ? x x ? ? 0 x x x x x x 1 ? 14 0 x x 2 1 x 1 0 x 2 2 x 0 1 1 ? 15 0 ? x x 0 x 0 x x x 0 x x 1 ? 16 0 x x 0 0 0 0 0 0 0 0 0 ? ? 17 0 ? ? ? 0 ?? ? 1 x x ? x ? 18 0 ? 0 0 x x 0 0 ? x x ? 19 0 ? 1 ?? 1 0 0 1 0 ? ? 20 0 ? x 0 x 0 ? 0 ? ? 21 0 ?? ? ? ? 0 ? ? ? 22 0 x ?? ?? ? x x ? 23 0 ? ? 0 1 1 ? 24 0 2 0 1 1 ? 25 0 ? 0 ? ? 26 0 ? ? ? 27 0 ? ? 28 0 ? 29 0
Here are some interesting combinations that I found:
( 4,28) ( 9,11) (12,14) (12,24) (14,17) (14,23) (14,24) (24,25)
--observer
The fact that this table of FAI failures is restricted to humourous scenarios doesn't seem to have made it much less realistic, though I'm not in much of a position to tell how realistic such a table really is. I considered creating a page about TheDarkerSideOfFAICriticalFailure?, which would have listed some worse scenarios, but there is plenty of "darkness" in the scenarios already listed here. If you don't want to know the details, you may want to skip the next paragraph.
For example in #4, think of how many books, movies, and video games are based on a lone (group of) hero(es) who struggle against nearly impossible odds to prevent an evil being from creating hell on earth. (This is why I didn't like "Lord Of The Rings", or "Buffy the Vampire Slayer". In these two examples, even if the heroes win, massive ammounts of suffering have already happened.) And for #24, what if the AI asked a fundamentalist christian "Do you really believe that the unbelievers and false christians deserve to go to hell for their sins?" Also, for any scenario involving wish-granting, remember that the emotion of hatred combined with the ability to have wishes granted results in some very bad scenarios.
I was recently told that discussing these scenarios was a bad idea because it just scares people without really accomplishing anything, except perhaps for shattering blind optimism, which sometimes needs to be shattered. However, I have found that the ability to calmly discuss these scenarios without being emotionally traumatized is extremely useful. Unfortunately, I do not claim to fully have this ability.
(Avoiding discussing these scenarios because they are so scary could be called "fearing fear itself", which, contrary to a popular saying, is sometimes a bad idea.)
--observer
Oof. I didn't even think of that, and I should have. This is a scenario which bears a superficial resemblance to "poetic justice", but which is so far beyond anything imaginable as deserved, that it ceases even to be poetic.
And before anyone posts to the wiki saying "What do you mean? It is poetic justice!", please ask yourself whether you'd say that to AI #24.
Scenario #22 is partially recoverable if the AI kept complete flight-recorder style logs; all of the simulated persons could be restored at a later date when the first Singularity historian examines the logs and notices that the simulations were sentient entities. Admittedly, it's still a lot of suffering.
For the critical success, how about 'Friendly Singularity that converges quickly to optimal'? Of course, no non-Power is intelligent enough to know what this means...
Hi, this is NathanRussell - I haven't applied for an account here yet. I'm pakaran on the chatroom, though (and almost everywhere else, I own the pakaran.org domain, among other things). To add to what Eliezer just said, keep in mind that Adolph Hitler, Jeffrey Dahmer, and most others that people would regard as evil inflicted a finite amount of suffering. A concept like hell inflicts an infinite amount of suffering, which means an amount beyond fifty years in prison, fifty to the power fifty years, or 50 -> 50 -> 50 -> 50 repeated enough times to fill this server, where the 50 are conway arrows, and the output is years of suffering. So it's possible to argue that the AI would be worse than every subjectively "evil" person in human history, combined. That's the stakes we're playing with when dealing with AI - grey goo isn't the worst scenario, nor is "grey goo trying to solve Goldbach" or any variations on that theme.
(just created an account, sorry about that -- NathanRussell)
That's the stakes with any form of different-than-human (especially absolutely smarter) intelligence, including all forms of uploading and human self-enhancement. "AI" typically implies minds "made from the ground up", which excludes uploads.
-- NickHay
Actually, it is still unknown whether or not the universe we live in is capable of an infinite amount of anything. It is also uncertain which is "TheConservativeAssumption?". It is also uncertain whether "TheConservativeAssumption?" is the assumption you "should" be making. The answer to the first question probably won't be known until after the Singularity, and the answers to the second two questions can change depending on the situation.
--observer
From the main GurpsFriendlyAI page;
> Thus the Friendly AI skill is harder to learn than the Seed AI skill, but can be > used in place of the Seed AI skill at +4... Teaching someone the Friendly AI skill > automatically implies teaching them the Seed AI skill.
In retrospect, this appears to be incorrect. Eliezer seems to be actively engaged in teaching people FAI without teaching them much seed AI, and -4 would be a more sensible default from FAI to seed AI than +4.
-- Starglider
Two extra scenarios I enjoy:
-- OlieNcLean?
Some new ideas:
-- Robin Lee Powell
Wouldn't the second case only be a failure if the AI concludes wrongly? If we are living in such a simulation, what would be the alternative?
-- Starglider
Another one:
Afficionados of violent computer games, be very, very afraid!
-- OlieNcLean?
Ah, but what about the ones who played with invincibility turned on...?
-- Random web surfer
-- Another random web surfer
Is this whole list obsolete now that SIAI isn't trying to create a FriendlyAI, but is instead trying to create an OptimizationProcess? that implements CoherentExtrapolatedVolition? --observer
Maybe, but everybody seems to enjoy it. For that matter, I was unaware of this OptimizationProcess?... Then again, that's typical. --AngelicaKlosky
All of these are possible outcomes of CEV, either because you made an error implementing it, or Just Because. The later scenario is theoretically not a critical failure, if you accept that CEV is 'right in principle' no matter what it produces. -- Starglider