From: Russell Wallace (russell.wallace@gmail.com)
Date: Mon Jan 09 2006 - 12:34:43 MST
On 1/9/06, Eliezer S. Yudkowsky <sentience@pobox.com> wrote:
>
> Okay, this is what I mean by an assumption that won't bear serious
> weight, if something bad happens to us when we're mistaken about it.
> Computations-performed-in-a-given-time isn't a calculable property of a
> fixed physical scenario, like the impact speed of a bowling ball dropped
> from 100 meters above the lunar surface. Superintelligences may search
> the entirety of actual physics, as it is known to them, seeking the
> tiniest loophole that will let them get away with computing faster.
> Maybe they won't find anything; maybe they will. Maybe they'll obey
> every relevant physical law you're thinking of, and yet, somehow, a lot
> of computation will get performed in what doesn't seem like much time.
> Human beings rise above the ground and fly between cities, but not by
> defying gravity.
>
*nods* I figured that was the sort of thing you meant. Now personally I'm
highly skeptical about the notion that loopholes in the Bekenstein bound,
for example, could be exploited merely by executing ADD and MOV instructions
on an array of Opteron chips - but I also think a plan that reads "First,
create a hostile superintelligence; second, assume it can't get around
such-and-such a set of safeguards" is in any case a bad one for quite
different reasons.
- Russell
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT