From: Russell Wallace (russell.wallace@gmail.com)
Date: Mon May 09 2005 - 21:39:55 MDT
On 5/10/05, Peter de Blanc <peter.deblanc@verizon.net> wrote:
> On Tue, 2005-05-10 at 02:55 +0100, Russell Wallace wrote:
> > Sure, but that's only one failure mode. What if the extrapolated
> > volition converges, but ends up leading to a region of state space
> > that doesn't contain sentient life?
>
> Well, if you're using a Last Judge, then he should catch that. I can see
> how relying upon a single human for the final decision could make you
> uncomfortable (indeed, it is a hack), but you are proposing that we rely
> on a committee of humans for all decisions, which seems even worse.
Oh no, it's not relying on a Last Judge I'm uncomfortable with; it's
not ideal, but it's better than a lot of possible alternatives.
The first problem is: _how does the Last Judge know what the course
currently contemplated will lead to_? The AI shows him? By what
algorithm is that vision computed? (Given that where complex
self-improving information processing systems are concerned there is
in general no faster way to predict the future than to implement it,
one can a priori suggest there won't be such an algorithm.)
The second problem: suppose you had an accurate enough algorithm, what
will the feedback loop of wish - show - change wish based on vision
produce? (My intuition for what it's worth is that the errors in the
algorithm will be quicker and easier to find than a genuinely good
future; cf. various genetic programming experiments.)
The third problem: if all humanity are really confined in the same
trajectory to bite at one another like rats in a box, what makes you
think all the nastier political instincts won't come out, complete
with finding the Last Judge most acceptable to the toughest and
nastiest faction?
The fourth, fifth and sixth problems are things I haven't thought of
yet. (What, you expected one human brain to think of _all_ the
problems in something that complicated, sensitive and dangerous? :))
How much do you want to bet none of them are deadly?
But suppose you can overcome all of those: Collective Volition
requires convergence, Domain Protection does not. So at the last I
will suggest: try CV then if you must, but keep DP ready as a fallback
in case the volition does not converge.
- Russell
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:56 MST