Re: Why playing it safe is the most dangerous thing

From: Ben Goertzel (
Date: Fri Feb 24 2006 - 10:33:44 MST

On 2/24/06, Peter de Blanc <> wrote:
> On Fri, 2006-02-24 at 06:04 -0500, Ben Goertzel wrote:
> > * If we launch a Singularity before the jerks in power figure out
> > what's up, we have a 50/50 or so chance of a good outcome (by the
> > Principle of Indifference, since what happens after the Singularity is
> > totally opaque to us lesser beings)
> But a 50/50 outcome is not the null hypothesis. If the Singularity is
> really totally opaque to us, then we should imagine that all universe
> states are equally probable post-Singularity. The vast majority of
> possible configurations of matter do not contain human life, so the null
> hypothesis is that humans almost certainly cease to exist.

But really, all these "null hypotheses" are just BS, right? I guess
we all understand that....

To pick a single counterargument to your argument: If the Singularity
is totally opaque to us, then why should we assume that probability
theory (which is ultimately an abstraction from human experience,
which is very limited in the grand sense) applies to the
post-Singularity world?

As a side point: After all, the general applicability of classical
probability theory is *already* in question even within the human
world, via Youssef's quantum probability theory...

> The Singularity is not totally opaque to us because we expect it to be
> the result of goal-oriented cognition, so we can expect the post-
> Singularity world to maximize some utility function.

I don't agree with this implication, which has merely been stated
rather than argued for

> I have a hard time believing that a randomly-selected utility function,
> when maximized, could result in human life, because humanity is very
> complex.

And I have a hard time accepting that this kind of reasoning, based on
human concepts abstracted from humans' limited experience, is
meaingfully applicable to the post-Singularity domain.

-- Ben G

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT