From: Nick Bostrom (nick.bostrom@philosophy.oxford.ac.uk)
Date: Sun Dec 11 2005 - 15:26:11 MST
Michael Wilson wrote:
>This proposal seems to me rather like Nick Bostrom's
>preffered option of building an 'Orcale'. This isn't
>as easy as it looks, even if you start from the position
>that it's pretty damn hard (sticking with the AGI norm
>then :) ), but I've been studying ways to do it and it
>does still look to me much easier than building any of
>Yudkowsky's FAI proposals.
("Preferred option" might be too strong. I don't have a settled opinion on
this matter.)
>An Oracle is of course still a tremendously dangerous
>thing to have around. Ask it for any kind of plan of
>action and you are allowing it to optimise reality to
>the maximum degree permitted by using you as intermediary,
>in effect bringing up a marginally diluted version of the
>'understanding intent' problem without any efforts to
>directly solve it (e.g. the way EV attempts to). I must
>reluctently classify Nick Bostrom's proposal to make an
>Oracle generally available (or at least, publically known
>and available to experts) as hopelessly naive. Clearly
>there is vast potential for misuse and abuse that would
>be unavoidable if publically known, at least in the short
>space of time before some fool asks one how to build a
>seed AI that will help them with their personal goals. It
>does seem likely to me that an Orcale built and used in
>secret, by sufficiently moral and cautious researchers,
>would be a net reduction in risk for an FAI project.
There are different options for who should decide what questions could be
posed to the Oracle. It might be difficult to ensure that the best such
option is instantiated. But this problem is not unique to the
Oracle-approach. It is also difficult to ensure that the first AGI is built
by the best people to do it. The question here is, for whichever group has
control over the first AGI - whether it's SIAI, the Pentagon, the UN, or
whatever - what is the best way to build the AGI?
The potential advantage with an Oracle is that it seems easier to specify
the goal to answer questions subject to certain simple constraints about
what changes may be brought about in the world to archive this objective
than to specify the goal to remake the world in accordance with "humanity's
extrapolated volition". Something like: Find the most accurate answer to
the question you can within 5 seconds by shuffling electrons in these
circuits and accessing these sources of information, and output the answer
in the form of 10 pages print-out.
If we could specify safe rules for avoiding the risk of turning the
universe into computronium and suchlike (rather crude failure modes), then
we could use the Oracle to ask for advise on various aspects on how best to
proceed with developing more ambitious AI if we want to, or we could just
ask for drugs to cure cancer etc. It seems nice to be able to take the
advise of a superintelligence about what would happen if we did various
things, rather than to immediately launch a potentially world-transforming
AI and hoping for the best.
Those who could ask the Oracle questions would be in a very powerful
position. The right to ask questions would obviously have to be limited
(don't want a terrorist asking what is the best way to destroy the
world...). But a wise asker could ask questions that could make future
steps safer. An unwise asker might ask dangerous questions, but presumably
the alternative is that they would build unsafe world-transforming AI
directly. And it might be easier to persuade users to ask a few wise
questions first, than to persuade them to build friendly world-transforming AI.
It takes pretty high level of responsibility to worry about the Last Judge
problem. If the askers are concerned with this problem, they could try
asking for advise on how to solve this problem. One could use the Oracle
without necessarily putting oneself in the Last Judge position.
Nick Bostrom
Director, Future of Humanity Institute
Faculty of Philosophy, Oxford University
10 Merton Str., OX1 4JJ, Oxford +44 (0)7789 74 42 42
Homepage: http://www.nickbostrom.com FHI: http://www.fhi.ox.ac.uk
For administrative matters, please contact my PA, Miriam Wood
+44(0)1865 27 69 34 miriam.wood@philosophy.ox.ac.uk
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:54 MDT