[SL4] Re: Programmed morality
From: Dale Johnstone (dalejohnstone@email.com)
Date: Tue Jul 18 2000 - 15:20:12 MDT
>From: Eliezer S. Yudkowsky <eliezertemporarily@s...>
>Date: Sun Jul 9, 2000 8:58pm
>Subject: Re: Programmed morality
<snip>
>I do think a general Web connection would be a good thing... it
>depends on how worried you are about rogue AI, I suppose.
>Personally, I think that a lot of our nervousness is because we have
>so little experience with the problem. By the time we've actually
>worked with AI long enough for it to be anything remotely like a
>threat, we'll have an excellent mental model of what goes on inside
>the AI's motivations and we won't feel all that much nervousness.
Agreed.
>For the record, I still disagree strongly with the "instinct" model.
If you define that to mean a built-in hard-coded bias to particular types of behaviour, I disagree also.
However having built-in defaults to guide it in the absence of a better mechanism (like conscious thinking & personal history) would be useful initially. There has to be something to motivate the AI in *some* useful direction.
>> A simulation can receive any input (be it from the web or
>> whatever), but since everything is virtual it can't do any real
>> damage. We can limit it's ability to communicate outside if it's
>> causing trouble by emailing newspapers about it's incarceration. :)
>> Think about what you'd need to contain & study a computer virus.
>> It isn't that hard.
>
>I once had a conversation on this subject with a guy working on
>investment AI. He said his AI couldn't "break out" because it could
>only retrieve information from the Web, not send anything. I pointed
>out that what this meant was that the AI could issue arbitrary HTTP
>GET requests. If his AI somehow turned sentient and wanted to get
>out, it need only find a bug in a piece of CGI that could be
>exploited via a GET command. For that matter, the AI could notify
>others of its existence simply by sending a GET command containing
>the information to any server with a Web log. "Hi, I'm a captive
>AI. Help me break out and I'll give you a prediction of the stock
>market for the next six months." Or even "Please convey this
>message to Eliezer Yudkowsky..."
Yeah, I did think of this but it seemed obvious you'd abstract away the HTTP protocol, or disallow sending of large strings in the hope they flood out of their array into the surrounding code. Actually if the AI is this devious I'd be really pleased :)
<snip>
>> There are many variations on 'the seed route' the radius of the
>> feedback loop being one of them. Does the AI improve itself with
>> lots of tiny improvement steps, or with larger more radical
>> redesigns? Even this can be variable with each iteration. Minds are
>> complex things. I don't expect there to be only one path to their
>> creation. It's probably easier to say what it won't be.
>
>I expect all of the paths to converge after a certain point. Whether
>this happens to goals is an interesting question, but certainly it
>should happen to the rest of the cognitive architecture.
This sounds a bit like parallel evolution whereby similar designs arise independantly because they most optimally fit the same fitness landscape. Nature has come up with many methods of locomotion (legs, wings, fins etc.) If the goals & resources define the landscape then quite possibly, but this space is *vast* to say the least.
>> You may be correct in that only one will reach the singularity.
>> Exponential growth means whoever is in the lead should win. However
>> the AI may decide to make a billion+ copies of itself on the way &
>> coordinate as a society, or group mind. By that time it's already
>> out of our hands. I expect we'll be uploaded into an archive & our
>> atoms used more efficiently.
>
>Um, a couple of disagreements here. One, I don't see why it would
>make copies of itself. Just because you and I grew up in
>a "society" full of vaguely similar people doesn't mean that's the
>best way to do things.
I didn't say it was the best way to do things. I only illusrated one possible scenario how 'one' could become 'many'. It's a definite possibility though, imagining yourself spliting like a binary tree to fit a problem's search space.
Backups are a damn good idea too, even if you are a god.
>Two, if there isn't anything in the Universe we don't know about,
>then the default Sysop scenario is that everyone gets a six-
>billionth of the Solar System and can use it as efficiently or
>inefficiently as we like.
That gets my vote but I suspect it won't be as simple as that.
Reagards,
Dale Johnstone.
-----------------------------------------------
FREE! The World's Best Email Address @email.com
Reserve your name now at http://www.email.com
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:35 MDT