Re:U.S. To Build Two Flagship Supercomputers for National Labs
by epistasis on Friday November 14, 2014 @04:39PM
(#8609650)
>The general reason comes down to inter-node bandwidth
And sometimes even more than bandwidth, inter-node latency, which is where Inifinband really shines over ethernet. Calculations which have high levels of node-to-node dependencies are pretty much the difference between super computers and the type of stuff that large internet companies compute on.
I hear (but don't personally know) that Google-style large data center installs are moving towards the CLOS-style networks that have been popular in HPC for a long time. These network topologies give equal bandwidth between any pair of nodes, as well as nearly equal latency.
Re:U.S. To Build Two Flagship Supercomputers for National Labs
by mscman on Friday November 14, 2014 @05:29PM
(#8609900)
Not just Google. Amazon's "Enhanced networking" uses a feature in 10GbE that's been used in smaller HPC clusters for around 5-10 years now. And MS Azure has InfiniBand backing their highest-tier instance types.
Many datacenters are adopting HPC technologies to reach the scale they need.
Things like physics simulations, weather etc., have a lot of interaction between any divisions you could try to draw to be able to split the information between nodes. To work with these problems you need a faster interconnect between processors than ethernet provides, infiniband seems a popular choice but I'm no expert on the details of their architecture. These problems also need to move in relative lockstep meaning loosely connected systems don't work as well or at least don't provide any real advantages. Node homogeneity makes these tightly coupled processing nodes easier to manage.
Occasionally there will be upgrades to a super computer but generally they're run 24/7 until they're too far behind the technology, both in speed and lower power consumption, that it becomes cheaper to completely replace it than to try to do an upgrade that will replace more than half the cluster's total electronics to begin with.