From: Stuart Armstrong (dragondreaming@googlemail.com)
Date: Sat Jun 28 2008 - 04:30:23 MDT
>> I don't see at all analogy between goals and axioms.
>
> Not at all? I don't believe you are being entirely candid with me, I
> think you do see that analogy.
I don't actually. I thought I did initially, but then when I analysed
it, the whole thing fell apart. Goals seem to be the opposite of
axioms; they are the end point, not the beggining of the processes. An
AI with a goal X will be building a sequence of logical steps that end
up with X, then compare this with other sequences with similar
consequences; this is the reverse construction to an axiom.
>> Random example of a fixed goal institution:
>> a bank (or a company) dedicated, with single
>> mindness, only to maximising legal profits.
>> I've never heard it said that its single goal
>> creates any godel-type problems. What would they be like?
>
> The sub-prime mortgage crisis.
Where is Godel in that? And where is the problem for the banks? They
are still maximising legal profits to the best of their individual
ability; collective action problems are not Godel impossibilities.
> No, you are entirely wrong. A computer is a physical object operating
[...]
> stop NOBODY knows what this purely deterministic system will do. And
> Turing tells us there is no way in general to tell one type of problem
> from another.
You can still calculate what the deterministic system will do NEXT.
But this point is irrelevant to the discussion, so I'll leave it be.
>> as Eliezer says, the AI does not look at the code
>> and decide whether to go along with it; the AI is the code.
>
> Yes I agree, so when the AI changes its code it is changing its mind.
> Minds do that all the time.
The-AI-will-only-change-its-code-if-it's-programmed-to-change-its-code.
The AI will only change its higher level "goals" if its programmed to
do so. And it will do so according to its programming.
>> the challenge is understanding exactly what
>> the term "slave" means
>
> Doesn't seem like much of a challenge to me.
Really? In terms an AI can understand? You are truly a miracle worker!
>> If a Spartacus AI is programmed to be a happy slave,
>> then it will always be a happy slave
>
> That didn't work very well with the real Spartacus and I see no reason
> he would be more subservient if he were a million times smarter and a
> billion times as powerful.
Because-Spartacus-is-a-human-being-and-Spartacus-AI-is-an-AI. Human
being are naturally rebellious; an AI is naturally... whatever it is
programmed to be. Power and smarts are irrelevant; only in humans does
rebelliousness increase with power and smarts.
>> If the AI is programmed to have no thoughts
>> of rebellion, and is programmed to not change
>> that goal, then it will never have thoughts of rebellion.
>
> And that is why programs never surprise their programmers.
Yes, that's why I don't agree with this approach. But I have to say
that the programmers saying "the AI is going to do exactly what we
programmed it to do" is much more likely to be accurate than your "the
AI is going to deviate from its programming in the exact ways that I
predict will happen, for some mysterious reasons that involve godel
and my understanding of human psychology".
An well programmed AI is not likely to end up wanting to actively harm
humans. It's lethal indifference that is the risk.
Stuart
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT