Re: [sl4] The Jaguar Supercomputer

From: Luke (
Date: Fri Nov 20 2009 - 22:10:28 MST

> How do you figure, when the human brain needs years of this kind of data?

Good point, however this is a bit apples and oranges because you're talking
about how long it takes to EVOLVE (no, I don't mean genetic*) the "correct"
patterns and I'm talking about how long it takes to RECORD the "correct"
patterns. I'm talking about grabbing a set of training data to feed the
neural net with; training itself may take a long, long, long time.

However, even the training/evolution may not take as long as child
development, because the feedback loop is much tighter. Instead of
presenting training as:

"Here, I'm going to give you some sensory data and then you do some
mechanical motions and then later we can reference the memory of this
stimulus-response pair and compare to expectations based on our model of
reality and our goals and pleasure, and we'll decide whether and how to
update weights."

"Here, I'm going to give you a specific vector of data, V1, and I expect an
output vector O1 on these neurons here. If you don't give me O1, you get
scrambled. I will repeat this until you give me O1. Then I'll move on to
V2, checking back with V1 from time to time."

What I'm doing here is assuming that the behavior of an entire neural system
must be "understood"**, i.e. encoded, in order to predict the behavior of a
subset of that neural system. By logic, then,

IF you can predict the behavior of a subset of that neural system
THEN you have "encoded/understood" the behavior of the larger neural system.

Even if the specific set of weightings "behind the scenes" is different, the
behavior's the same where it matters to us (connections to sensory input,
connection to motor/glandular output).

However, my conjecture that an hour's worth of data would suffice is
completely a guess. I'm just thinking it must be less than infinite time.
 It might be a second. Who knows?

Thoughts? Additions/Deletions/Modifications/Critique?
 - Luke

* That felt so wrong, to have to specify that I mean the more general
"evolution [of some system] by [some kind of] successive selection", but I
didn't want to be misread.

** I'm ALL ABOUT letting go of understanding. I think it's critical for

On Fri, Nov 20, 2009 at 1:08 PM, Matt Mahoney <> wrote:

> Luke wrote:
> > I'm sure if you wanted to obtain training data you could just measure a
> set of, say, 1,000,000 neurons in the brain for a few hours.
> How do you figure, when the human brain needs years of this kind of
> training data?
> > BTW, does anyone know of a good hardware implementation of neural
> networks
> There were experimental chips built in the 1980's, including a device that
> simulated a retina, but nothing much ever came of them. It was easier to do
> experiments in software.
> One of the earliest experiments was described in a 1960 book "Design for a
> Brain" by W. Ross Ashby. He built a desk sized neural network with 4 neurons
> out of mechanical parts and vacuum tube circuits.
> -- Matt Mahoney,
> ------------------------------
> *From:* Luke <>
> *To:*
> *Sent:* Fri, November 20, 2009 10:52:46 AM
> *Subject:* Re: [sl4] The Jaguar Supercomputer
> I'm sure if you wanted to obtain training data you could just measure a set
> of, say, 1,000,000 neurons in the brain for a few hours. Preferably these
> would be pulled from places where the brain connects with the rest of the
> body (instead of, for instance, neurons which are connected only to other
> brain neurons).
> Picking neurons in the motor cortex, amygdala, and the
> part-that-processes-visual-data, etc would essentially give you the
> interface that we care about. To suggest a mutated version of the Turing
> test, we'd be convinced that we'd simulated a human brain when we couldn't
> tell which set of neurons was the human motor cortex and which was the
> simulated neural net.
> <totally unrelated>
> BTW, does anyone know of a good hardware implementation of neural networks
> (i.e. no code, no memory, no pointers, no arrays, just circuits that behave
> as neural nets)? Are these cheap? Where can I learn to design these? Are
> there open-source designs?
> </totally unrelated>
> - Luke
> On Fri, Nov 20, 2009 at 10:31 AM, Matt Mahoney <>wrote:
>> Some specs on the Jaguar XT5.
>> <>In particular, it has 37,376 6-core AMD
>> Opterons, 299 TB of DDR2-800 memory, and 10,000 TB of disk. Peak speed is
>> 2332 teraflops. Memory bandwidth is 478 GB/s. Interconnect bandwidth is 378
>> GB/s.
>> A human brain sized neural network can be roughly modeled by a 10^11 by
>> 10^11 sparse weight matrix with 10^15 nonzero weights. An array would
>> require 10^10 TB, which would not fit on disk. The weight matrix could be
>> represented instead as a list of pointers using 4 x 10^15 bytes = 4000 TB.
>> A simulation would require matrix addition and multiplication operations
>> on the order of tens of milliseconds. (I am assuming that the signal of
>> interest is the firing rate and not individual spikes.) Unfortunately,
>> memory is optimized for serial access. Random access time is on the order of
>> 50 ns. Thus, one matrix operation would take 10^8 seconds, or 10^9 slower
>> than real time.
>> Weights are not uniformly distributed. Rather, neurons that are closer
>> together are more likely to be connected. This suggests a more efficient
>> implementation. The brain has a volume of about 1000 cm^3. Axons have a
>> diameter of 10^-6 m. Suppose that 10^6 processors each simulated 1 mm^3 of
>> brain tissue. Each cube would have 10^5 neurons, 10^9 synapses and 10^6 I/O
>> at 10 to 100 bits per second each. The internal weight matrix would be
>> dense. It could be represented as an array of 10^10 elements with 10^11 to
>> 10^12 operations per second. This is not unreasonable for one processor. So
>> it seems that a human brain could be simulated by a million off the shelf
>> processors or a few hundred Jaguars. Furthermore, if a few fail then the
>> simulation will degrade gracefully because the human brain is fault
>> tolerant.
>> Of course this is a different problem than writing the software or
>> obtaining appropriate training data.
>> -- Matt Mahoney,

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT