# Re: Is a theory of hard take off possible? Re: Investing in FAI research: now vs. later

From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Feb 20 2008 - 20:52:10 MST

William Pearson wrote:
>
> Now imagine you added a link to the outside, through which one bit
> could enter. Now depending upon that bit the system, the systems
> evolution could bifurcate, or it could ignore it and stay singular.
> Bifurcation leads to the potential for growing of the ability to
> predict the world, if parts of the world happen to correlate with the
> bit that was bifurcated on.
>
> More bits given to the system lead to more possible bifurcations.
> Exponentially increasing numbers of bifurcations are needed for
> exponential increases in predictive power.

You need a logarithmic number of bits in order to slice a prediction
to great fineness - i.e., predicting to within a factor of a billionth
requires 30 bits, not a billion bits. I know this is probably what
you meant, but nobody would hear you saying it unless they already

The other thing to remember is that we do not live in a random
universe where it takes 1 bit of information to predict one more bit
of an important outcome, like trying to record a sequence of
coinflips. Gravity is the same all over, you don't have to learn it
again each time; and many important and manipulable aspects of the
universe are highly regular.

> A concrete example, let us say we are trying to get an AI to evolve to
> a state where it can consistently use my IP number to reach my
> computer. It follows its programming, let us say by incrementing a
> counter and using that in the IP field of TCP/IP packets. I give it
> no feedback nor any other bits of information about the world. It will
> never be able to bifurcate to a state where it always uses the correct
> IP address, it is just throwing out packets into the void. At some
> points it would even get the IP address right, but it wouldn't know

> Now say I give it a bit per second bandwidth input. If it has used the
> correct IP address in the previous second it gets a 1, otherwise a 0.
> Now it might guess the correct IP address first time (and only guess
> one IP address) and get a 1, and therefore get 4 bytes of information.
> But as we said the AI is deterministic and the IP address is fixed, so
> we might be forgiven in thinking that the game is rigged and the
> system already had that information. If we randomise the IP and set it
> off again, on average, no matter the computing power the system had,
> it would get only 7.45x10^-9 bits per second of the IP address. This
> isn't the most efficient coding of our input to it. We could simply
> give it the IP address, one bit per second. The system would still be
> limited to bifurcating once per second.

You are going down a correct path, mathematically speaking; but the
answer is going to end up being "In principle, it takes a ridiculously
small amount of information; the exact amount is theoretically
incomputable; and a superintelligence would require more, but we can't
guess how much more without actually running a superintelligence."

A cable modem connected to the Internet is certainly *vastly* more
bandwidth than is theoretically needed to grok and take over the
world, though. Or a 1200 baud modem, for that matter. Hell, give me
a millionfold subjective speedup and a few siderial days to think
about it and *I'll* take over the world with a 1200 baud modem (not
because I want it; just to demonstrate the point).

Remember that there are more reliable ways of making money than trying
to win the lottery - a wise agent can steer their pressure to the more
manipulable, more predictable, lower-entropy levers in the system.
When bifurcation is a limited resource, you can arrange to use less of
it - you don't *have* to let 50% of your selves emit alternate
versions of every bit you want to output.

```--
Eliezer S. Yudkowsky                          http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
```

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT