SITE: "Friendly AI" appears at last!

From: Eliezer S. Yudkowsky (
Date: Wed Apr 18 2001 - 11:45:03 MDT

The Singularity Institute is pleased to announce that the "open
commentary" version of the long-awaited "Friendly AI" is now available to
the academic and futurist communities.

Complete table of contents:

    Preface [2K]
    INIT [1K]
    4.1: Design requirements of Friendliness [50K]
        4.1.1: Envisioning perfection
        4.1.2: Assumptions "conservative" for Friendly AI
        4.1.3: Strong Singularity, seed AI, the Transition Guide...
        4.1.4: Goal-oriented behavior
    Interlude: The story of a blob [12K]
    4.2: Humans, AIs, and SIs: Beyond anthropomorphism [78K]
        4.2.1: Reinventing retaliation
        4.2.2: Selfishness is an evolved trait
   Pain and pleasure
       FoF: Wireheading 1
   Anthropomorphic capitalism
   Mutual friendship
   A final note on selfishness
        4.2.3: Observer-biased beliefs evolve in
               imperfectly deceptive social organisms
        4.2.4: Anthropomorphic political rebellion is just plain silly
        Interlude: Movie cliches about AIs
        4.2.5: Review of the AI Advantage
    Interlude: Beyond the adversarial attitude [17K]
    4.3: Design of Friendship systems [0K]
        4.3.1: Generic goal systems [78K]
   Generic goal system functionality
   Layered mistake detection
       FoF: Autonomic blindness
   FoF: Non-malicious mistake
       Anthropomorphic injunctions
       Adversarial injunctions
       AI injunctions
   Ethical injunctions
       Anthropomorphic ethical injunctions
       AI ethical injunctions
   FoF: Subgoal stomp
   Emergent phenomena in generic goal systems
       Convergent subgoals
       Anthropomorphic satisfaction
        4.3.2: Seed AI goal systems [105K]
   Equivalence of self and self-image
   Coherence and consistency through self-production
       Look-ahead: Coherent supergoals
   Programmer-assisted Friendliness
       Unity of will
       Cooperative safeguard: "Preserve transparency"
       Absorbing assists into the system
       Programmer-created beliefs must be truthful...
   Wisdom tournaments
       Wisdom tournament structure
   FoF: Wireheading 2
   Directed evolution in goal systems
       Anthropomorphic evolution
       Evolution and Friendliness
       Conclusion: Evolution is not safe
   FAI hardware: The flight recorder
        Interlude: Why structure matters [7K]
        4.3.3: Friendly goal systems [4K]
   External reference semantics [67K]
       Bayesian sensory binding
       External objects and external referents...
           Flexibility of conclusions...
       Bayesian reinforcement
           Bayesian reinforcement...
           Perseverant affirmation...
       Bayesian programmer affirmation...
   Shaper/anchor semantics [59K]
       "Travel AI": Convergence begins to dawn
       Some forces that shape Friendliness
       Beyond rationalization
       Shapers of philosophies
           SAS: Correction of programmer errors
           SAS: Programmer-independence
           SAS: Grounding for ERS...
           Positive anchors
           Negative anchors
           Anchor abuse
       Shaper/anchor semantics require intelligence...
   Causal rewrite semantics [37K]
       The physicalist explanation of Friendly AIs
       Causal rewrites and extraneous causes
       The rule of derivative validity
       Truly perfect Friendliness
       The acausal level
   The secret actual definition of Friendliness [8K]
       Requirements for "sufficient" convergence
        4.3.4: Developmental Friendliness [28K]
   Teaching Friendliness content
       Trainable differences for causal rewrites
   Commercial Friendliness and research Friendliness
       When Friendliness becomes necessary
       Evangelizing Friendliness
   "In case of Singularity, break glass"...
       The Bayesian Boundary
       Controlled ascent
    Interlude: Of Transition Guides and Sysops [10K]
        The Transition Guide
        The Sysop Scenario
    4.4: Policy implications [34K]
        4.4.1: Comparative analyses
   FAI relative to other technologies
   FAI relative to computing power
   FAI relative to unFriendly AI
   FAI relative to social awareness
   Conclusions from comparative analysis
        4.4.2: Policies and effects
   Regulation (-)
   Relinquishment (-)
   Selective support (+)
        4.4.3: Recommendations
    4.5: Miscellaneous [4K]
        4.5.1: Relevant literature
    END [1K]
    Appendix 4.A: Friendly AI Guides and References [0K]
        4.A.1: Indexed FAQ [27K]
        4.A.2: Complete Table of Contents [0K]

This is not the official launch of Friendly AI; this is the "open
commentary" version we're circulating in the community first. However,
you are politely requested to check the Indexed FAQ before sending in your
commentary, since we've already heard quite a few questions about Friendly
AI, and your comments may have already been taken into account.

-- -- -- -- --
Eliezer S. Yudkowsky
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT