From: Peter C. McCluskey (pcm@rahul.net)
Date: Tue May 24 2005 - 16:40:08 MDT
sentience@pobox.com (Eliezer S. Yudkowsky) writes:
>Peter C. McCluskey wrote:
>> If CV is the right approach to Friendliness, then no. The cpu time
>> needed to do CV right appears sufficient to delay development enough to
>> make it irrelevant.
>> Which is another reason to aim for making the first AI an obedient AI,
>> even if we need Friendly AI soon after that.
>
> From the PAQ list in _Collective Volition_:
>
>**
>
>Q10. How accurate and detailed does the extrapolation of collective volition
>need to be in order to work? (Frequently Asked)
The ambiguous answer to that hardly explains much. But my objection was
a bit hasty and based on an imperfect recollection of your writings.
In rereading _Collective Volition_, I came across this bizarre claim
that appears to indicate the core of what you and I disagree about:
Q7. Where is the collective volition getting this kind of power?
(Frequently Asked)
A7. A Really Powerful Optimization Process is not a god. A Really
Powerful Optimization Process could tear apart a god like tinfoil.
If you expect the RPOP to become this powerful within a few years of
being started, then your prediction is so many orders of magnitude
away from what most intelligent people who have thought about that
kind of question expect that you shouldn't expect to be taken seriously
without a very carefully phrased argument. Yet it's hard to see any
argument that is serious enough to be worth responding to. Eric Baum
makes some moderately strong arguments in his book What is Thought?
against your claim. If your plans depend on a hard takeoff and your
reasons for expecting a hard takeoff are no better than the ones I've
run across, then I'm pretty certain you will fail.
What kind of goal system do you plan to have built into the RPOP while
it is computing CV? Presumably you see the risk that it will take over
the world before you get around to using any results from the CV
simulations. Yet I can't see any indication of how well you would be
able to handle this risk.
It would also be nice to know how the RPOP's goals will enable or prevent
it being used to deal with a nonfriendly AI should one be developed before
the CV simulation produces any results.
-- ------------------------------------------------------------------------------ Peter McCluskey | Everyone complains about the laws of physics, but no www.bayesianinvestor.com| one does anything about them. - from Schild's Ladder
This archive was generated by hypermail 2.1.5 : Tue Feb 21 2006 - 04:22:56 MST