News for hackers. Stuff that matters.
Re:Deep Learning With Commodity Off-The-Shelf High Performan... by nkurz on Monday June 17, 2013 @09:01PM (#5896939)
Abstract:

  Scaling up deep learning algorithms has been
  shown to lead to increased performance in
  benchmark tasks and to enable discovery of
  complex high-level features. Recent efforts
  to train extremely large networks (with over
  1 billion parameters) have relied on cloud-
  like computing infrastructure and thousands
  of CPU cores. In this paper, we present tech-
  nical details and results from our own system
  based on Commodity Off-The-Shelf High Performance
  Computing (COTS HPC) technology:
  a cluster of GPU servers with Infiniband 
  interconnects and MPI. Our system is able to 
  train 1 billion parameter networks on just 3 machines
  in a couple of days, and we show that it can scale
  to networks with over 11 billion parameters using
  just 16 machines.  As this infrastructure is much
  more easily marshaled by others, the approach 
  enables much wider-spread research with extremely
  large neural networks.
For $20,000, they were able to build a 1-billion-connection system comparable to the $1MM system they built the previous year. Also in this paper, Andrew Ng and others detail how for $100,000 they also created an 11-billion-connection deep learning system with 16 commodity servers, each loaded with 4 Nvidia GTX680 GPU cards.
Re:Deep Learning With Commodity Off-The-Shelf High Performan... by cr4zy on Monday June 17, 2013 @11:18PM (#5897322)
This means that to emulate the near one quadrillion connections in the human brain would take about $10B today, assuming they can scale this system by 5 orders of magnitude with the same connections per dollar. Although they did manage to double their connections per dollar going from 1 to 11 billion connections.
Re:Deep Learning With Commodity Off-The-Shelf High Performan... by Udo on Tuesday June 18, 2013 @07:44AM (#5898599)
> This means that to emulate the near one quadrillion connections in the human brain would take about $10B today

It's important though to recognize that we're still missing the appropriate software to run brains. Technically, BlueBrain can run a cat's cortex today. Practically though just pumping up the number of neurons and connections does not yield general intelligence or consciousness, so BlueBrain cannot actually run a cat. Just as an even bigger supercluster cannot just run a human-like intelligence, even though the processing power might be there.

Furthermore, these are brute-force simulation ambitions. I would go out on a limb here and assert that we could actually run a human-level consciousness on much less hardware. Computers are very good at some things where our brains have to mount enormous infrastructure just in order to do OK.

Re:Deep Learning With Commodity Off-The-Shelf High Performan... by duaneb on Tuesday June 18, 2013 @02:55AM (#5897949)
I would assume that at some point latency would be a larger issue.
Re:Deep Learning With Commodity Off-The-Shelf High Performan... by waps on Tuesday June 18, 2013 @03:57AM (#5898097)
The great thing about pulse-based neural networks (like a human brain) is that latency, as long as it's constant, is not an issue. In fact you'd want connections with a wide variety of different latencies. They should be random, but remain constant over the life of the simulation.

Plus you have to keep in mind that this is a simulation. Computer can't keep up ? Run it at half "real-time".