**From:** Eliezer Yudkowsky (*sentience@pobox.com*)

**Date:** Thu Jun 03 2004 - 11:50:57 MDT

**Next message:**Eliezer Yudkowsky: "Re: FAI: Collective Volition"**Previous message:**Metaqualia: "Re: Sentiencee [Was FAI: Collective Volition]"**In reply to:**Norm Wilson: "RE: another objection"**Next in thread:**Metaqualia: "Re: another objection"**Reply:**Metaqualia: "Re: another objection"**Reply:**Damien Broderick: "Re: another objection"**Reply:**Thomas Buckner: "Re: another objection"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Norm Wilson wrote:

*>
*

*> Because morality is an abstract concept that affects human behavior, but
*

*> is not itself physically measurable by the FAI. The FAI cannot (so far
*

*> as we know) directly "perceive" morality, so it considers humans to be
*

*> the only available measuring devices and assumes that smarter humans who
*

*> know more are better at measuring (or at least describing, or behaving
*

*> in accordance with) the concept of morality. To remove humans from the
*

*> process would be analogous the throwing out the thermometer and
*

*> extrapolating the current temperature based on past results. By
*

*> teaching us more, the FAI would effectively be turning us into better
*

*> "morality thermometers".
*

It would be more accurate to say that an FAI needs to look at humans to

find the specification of how to compute morality. It's like a sentient

calculator that wonders "What is the Multiplication?" where the

Multiplication turns out to consist of multiplying 6 x 7. The Calculator

is pretty sure that the answer is somewhere in the vicinity of 41 or 43,

although different Calculators have been known to disagree on the subject.

But certainly a superintelligence would know that the Multiplication was

41, or 43, or whatever; not return preposterous answers such as 4 or 30,000.

Unless the superintelligence happened to contain a question, "What is the

Multiplication?", that consisted of multiplying 2 x 2 or 300 x 100.

If you presented this scenario to Calculators who did not quite understand

how Multiplication worked, they would laugh at the idea that a

superintelligence could get the answer so blatantly wrong - thinking that 4

or 30,000 was the Multiplication! The Calculators may not really

understand how Multiplication works, or whether the answer is 41 or 43, but

they know a superintelligence wouldn't be that stupid. Even if a

superintelligence started out with a Multiplication of 4 or 30,000, surely

it would look over this Multiplication, realize how silly and inappropriate

it was, and choose a better Multiplication like, oh, say, 45 or 39; it

might seem a little alien, but not something grossly wrong like 4 or

30,000. Would not the superintelligence realize the meaninglessness of 4,

that it cannot possibly result from multiplying 6, let alone 7? Maybe the

superintelligence would multiply 6 by 2/3, but that seems implausible.

Note that all Calculators, if they knew more, thought faster, were more the

people they wished they were, had grown up farther together, would

uniformly agree that the Multiplication was exactly 42; and in this sense

their Multiplication is as objective as mathematics. The Calculators'

mistake is to think that the superintelligence's Multiplication must

necessarily ask the same implicit question.

I now quote Damien Broderick and Adrian Tymes from a recent discussion on

the Extropians list:

Damien Broderick wrote:

* >
*

* > It's a *super-intelligence*, see, a constructed mind that
*

* > makes the puny human *insignificant* by comparison--so what *else* is it
*

* > going to do except get trapped immediately by a really dumb pun into
*

* > turning the cosmos in smiley faces?
*

Adrian Tymes wrote:

* >
*

* > [...]
*

* > the capacity for SIs to overcome their optimization
*

* > functions and decide on new ones - for example, the
*

* > paperclip maximizer who would realize that paperclips
*

* > only have meaning if there's something for them to
*

* > clip, and other sentient units for the convenience of
*

* > a clip to serve. (Unless you propose that a SI would
*

* > not strive to understand why it does what it does,
*

* > which would seem to strongly interfere with any
*

* > capability for self-improvement.)
*

Funny how natural selection hasn't picked a different optimization

criterion than reproductive fitness.

-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence

**Next message:**Eliezer Yudkowsky: "Re: FAI: Collective Volition"**Previous message:**Metaqualia: "Re: Sentiencee [Was FAI: Collective Volition]"**In reply to:**Norm Wilson: "RE: another objection"**Next in thread:**Metaqualia: "Re: another objection"**Reply:**Metaqualia: "Re: another objection"**Reply:**Damien Broderick: "Re: another objection"**Reply:**Thomas Buckner: "Re: another objection"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:47 MDT
*