From: Joshua Fox (joshua@joshuafox.com)
Date: Wed Jun 28 2006 - 08:31:21 MDT
Yudkowsky posed an urgent ethical challenge at the end of his Summit speech:
Work to bring on a Friendly Singularity, or enjoy the conference and return
to one's television.
This brought to mind some fascinating pieces by Carl Sagan and Douglas
Hofstadter which I read during the 1980's. These two scientists are today
widely admired by Singularitarians for cross-disciplinary interests,
scientific creativity, and compelling exposition, as well as ideas directly
related to the Singularity; during the Cold War they wrote eloquently
(including some very interesting scientific arguments) on the ethical
mandate to work for nuclear disarmament.
As Anissimov points out (in "Who are Technological Singularity Activists?"),
humanity first knowingly faced extinction during the Cold War. Scientists,
and especially nuclear scientists, were among the first to warn of the
danger, and then to actively work to avert it, just as computer specialists
are the most Singularity-aware today. Yet the species-death risk of that
scenario suddenly went way down. (Of course, many existential risks still
exist, including nuclear destruction.)
Analogical arguments are always imperfect, but if the analogy holds, what
conclusions can we draw?
1. Paradoxically, the most seemingly logical solution might actually make
the situation worse. Robert Aumann, who won the Nobel Prize in Economics for
game-theory work developed partly to analyze the Cold War, has been quoted
as saying that "peace was maintained because airplanes carrying nuclear
weapons were in the air 24 hours a day". Though I must, even in retrospect,
admire and support Dr. Rotblat over Dr. Strangelove, any reduction in the
MAD balance of terror might have actually *increased *the probability of
total nuclear war.
2. Therefore, though I also admire and support those who work to avert
Singularity disaster and to bring a Friendly Singularity, an
uncertainty-weighted cost/benefit analysis based on this analogy suggests
that one need not devote time and money to the Friendly Singularity, just as
most people who supported human survival did not give resources to the
disarmament movement.
Anissimov warns (in "Who Cares About the Singularity?") against responding
to the Singularity simply as a fact about the future, rather than as a call
to action. Yet even Kurzweil has said (in The Singularity Is Near) that
since we cannot know how to create Friendliness, we should focus on
enhancing democracy, human rights, progress, and prosperity (good advice
back during the Cold War, too), and hope for the best.
Inaction based on this sort of uncertainly could be all too common, even
among people whom the Singularitarian movement might want as supporters. How
might Singularitarianism deal with this?
Joshua
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT