From: Matt Mahoney (matmahoney@yahoo.com)
Date: Sat Nov 21 2009 - 13:21:06 MST
I understand your approach now. You record a sequence of inputs and outputs, then search for an internal state that produces the same outputs.
I think you need to be smart about selecting the inputs so that you test every memory location. Think about the problem of reverse engineering the state of a computer running unknown software.
Also, for a human brain you need to collect at a minimum 10^9 bits of compressed output (using Landauer's estimate of human long term memory) or 10^15 bits (counting synapses).
-- Matt Mahoney, matmahoney@yahoo.com
________________________________
From: Luke <wlgriffiths@gmail.com>
To: sl4@sl4.org
Sent: Sat, November 21, 2009 12:10:28 AM
Subject: Re: [sl4] The Jaguar Supercomputer
> How do you figure, when the human brain needs years of this kind of data?
Good point, however this is a bit apples and oranges because you're talking about how long it takes to EVOLVE (no, I don't mean genetic*) the "correct" patterns and I'm talking about how long it takes to RECORD the "correct" patterns. I'm talking about grabbing a set of training data to feed the neural net with; training itself may take a long, long, long time.
However, even the training/evolution may not take as long as child development, because the feedback loop is much tighter. Instead of presenting training as:
"Here, I'm going to give you some sensory data and then you do some mechanical motions and then later we can reference the memory of this stimulus-response pair and compare to expectations based on our model of reality and our goals and pleasure, and we'll decide whether and how to update weights."
"Here, I'm going to give you a specific vector of data, V1, and I expect an output vector O1 on these neurons here. If you don't give me O1, you get scrambled. I will repeat this until you give me O1. Then I'll move on to V2, checking back with V1 from time to time."
What I'm doing here is assuming that the behavior of an entire neural system must be "understood"**, i.e. encoded, in order to predict the behavior of a subset of that neural system. By logic, then,
IF you can predict the behavior of a subset of that neural system
THEN you have "encoded/understood" the behavior of the larger neural system.
Even if the specific set of weightings "behind the scenes" is different, the behavior's the same where it matters to us (connections to sensory input, connection to motor/glandular output).
However, my conjecture that an hour's worth of data would suffice is completely a guess. I'm just thinking it must be less than infinite time. It might be a second. Who knows?
Thoughts? Additions/Deletions/Modifications/Critique?
- Luke
* That felt so wrong, to have to specify that I mean the more general "evolution [of some system] by [some kind of] successive selection", but I didn't want to be misread.
** I'm ALL ABOUT letting go of understanding. I think it's critical for progress.
On Fri, Nov 20, 2009 at 1:08 PM, Matt Mahoney <matmahoney@yahoo.com> wrote:
Luke wrote:
>> I'm sure if you wanted to obtain training data you could just measure a set of, say, 1,000,000 neurons in the brain for a few hours.
>
>
>How do you figure, when the human brain needs years of this kind of training data?
>
>
>> BTW, does anyone know of a good hardware implementation of neural networks
>
>
>There were experimental chips built in the 1980's, including a device that simulated a retina, but nothing much ever came of them. It was easier to do experiments in software.
>
>
>One
> of the earliest experiments was described in a 1960 book "Design for a Brain" by W. Ross Ashby. He built a desk sized neural network with 4 neurons out of mechanical parts and vacuum tube circuits.
>
>> -- Matt Mahoney, matmahoney@yahoo.com
>
>
>
>
>
________________________________
From: Luke <wlgriffiths@gmail.com>
>To: sl4@sl4.org
>Sent: Fri, November 20, 2009 10:52:46 AM
>Subject: Re: [sl4] The Jaguar Supercomputer
>
>
>>I'm sure if you wanted to obtain training data you could just measure a set of, say, 1,000,000 neurons in the brain for a few hours.
> Preferably these would be pulled from places where the brain connects with the rest of the body (instead of, for instance, neurons which are connected only to other brain neurons).
>
>
>Picking neurons in the motor cortex, amygdala, and the part-that-processes-visual-data, etc would essentially give you the interface that we care about. To suggest a mutated version of the Turing test, we'd be convinced that we'd simulated a human brain when we couldn't tell which set of neurons was the human motor cortex and which was the simulated neural net.
>
>
><totally unrelated>
>BTW, does anyone know of a good hardware implementation of neural networks (i.e. no code, no memory, no pointers, no arrays, just circuits that behave as neural nets)? Are these cheap? Where can I learn to design these? Are there open-source designs?
></totally unrelated>
>
>
> - Luke
>
>
>
>
>
>
>On Fri, Nov 20, 2009 at 10:31 AM, Matt Mahoney <matmahoney@yahoo.com> wrote:
>
>>>Some specs on the Jaguar XT5. http://www.nccs.gov/jaguar/
>>In particular, it has 37,376 6-core AMD Opterons, 299 TB of DDR2-800 memory, and 10,000 TB of disk. Peak speed is 2332 teraflops. Memory bandwidth is 478 GB/s. Interconnect bandwidth is 378 GB/s.
>>
>>
>>A human brain sized neural network can be roughly modeled by a 10^11 by 10^11 sparse weight matrix with 10^15 nonzero weights. An array would require 10^10 TB, which would not fit on disk. The weight matrix could be represented instead as a list of pointers using 4 x 10^15 bytes = 4000 TB.
>>
>>
>>A simulation would require matrix addition and multiplication operations on the order of
>> tens of milliseconds. (I am assuming that the signal of interest is the firing rate and not individual spikes.) Unfortunately, memory is optimized for serial access. Random access time is on the order of 50 ns. Thus, one matrix operation would take 10^8 seconds, or 10^9 slower than real time.
>>
>>
>>Weights are not uniformly distributed. Rather, neurons that are closer together are more likely to be connected. This suggests a more efficient implementation. The brain has a volume of about 1000 cm^3. Axons have a diameter of 10^-6 m. Suppose that 10^6 processors each simulated 1 mm^3 of brain tissue. Each cube would have 10^5 neurons, 10^9 synapses and 10^6 I/O at 10 to 100 bits per second each. The internal weight matrix would be dense. It could be represented as an array of 10^10 elements with 10^11 to 10^12 operations per second. This is not unreasonable for one processor. So it seems that a human brain could be simulated by a
>> million off the shelf processors or a few hundred Jaguars. Furthermore, if a few fail then the simulation will degrade gracefully because the human brain is fault tolerant.
>>
>>
>>Of course this is a different problem than writing the software or obtaining appropriate training data.
>>
>> -- Matt Mahoney, matmahoney@yahoo.com
>>
>>
>
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT