Things like physics simulations, weather etc., have a lot of interaction between any divisions you could try to draw to be able to split the information between nodes. To work with these problems you need a faster interconnect between processors than ethernet provides, infiniband seems a popular choice but I'm no expert on the details of their architecture. These problems also need to move in relative lockstep meaning loosely connected systems don't work as well or at least don't provide any real advantages. Node homogeneity makes these tightly coupled processing nodes easier to manage.
Occasionally there will be upgrades to a super computer but generally they're run 24/7 until they're too far behind the technology, both in speed and lower power consumption, that it becomes cheaper to completely replace it than to try to do an upgrade that will replace more than half the cluster's total electronics to begin with.
And sometimes even more than bandwidth, inter-node latency, which is where Inifinband really shines over ethernet. Calculations which have high levels of node-to-node dependencies are pretty much the difference between super computers and the type of stuff that large internet companies compute on.
I hear (but don't personally know) that Google-style large data center installs are moving towards the CLOS-style networks that have been popular in HPC for a long time. These network topologies give equal bandwidth between any pair of nodes, as well as nearly equal latency.
Many datacenters are adopting HPC technologies to reach the scale they need.
Scaling.
A very easy out for me to explain is imagine if you could only simulate two molecules in a Molecular Dynamics simulation and both had to have 100 atoms or less. There's plenty of opportunity, sure. But what if you wanted to simulate an entire strand of RNA interacting with a protein? There are techniques and stable approximations to avoid the O(n^2) complexity of a molecular dynamics simulation but the next-generation hardware allows you to run the same simulations with the same limitations at a much bigger scale, or the previous-sized problem without as many assumptions. edit: There are other considerations of course; I am vastly oversimplifying with this example.
Also, the more simulations you can run the more uncertainties you can account for in Monte Carlo computations, so models can evolve to contain more details and less assumptions.
adding response to: > That way instead of doing a "big bang" upgrade such as this, you just upgrade individual nodes as the technology allows and are almost always "current."
The problem is that programming targeting a supercomputer usually involves taking advantage of specific features of the hardware to get the bleeding edge performance. Mixing and matching hardware quickly makes deployment of the code difficult in addition to writing the code to reliably squeeze every ounce out of every chip.
Now, suppose your a scientist who want's to simulate a hurricane. You create a model for the first step (second, minute, or nanosecond) and can send that to a computer, but what do you send to the next computer. It needs results from the first simulation to run, but in that case you might as well just keep running the simulation on the same computer. Clearly to use a cluster your going to need to break up your simulation into pieces and send each part to a separate computer. But, now after each step all the nodes need to talk to all the other nodes to keep the simulation in sync. Worse the more chunks you separate the problem into the more parts need to be talking to other parts.
This is also why supercomputers tend to have equal strength nodes. Having one node finish before the other is pointless as it needs results from other nodes to continue. In the end you need not just massive bandwidth but also low latency. If a step takes 1/1000th of a second to run but then needs to wait a full second to run the next step then your simulation get's 1 step per second not per 1/1000th of a second.
You also want to back up each node so if one of them crashes you don't need to re run the simulation from scratch.
That way instead of doing a "big bang" upgrade such as this, you just upgrade individual nodes as the technology allows and are almost always "current."
PS - I find their usage of the term "energy independence" hilarious. That was a term coined largely to justify fracking and other environmentally damaging practices in the US. I'm glad to see it has been over-used so much now third parties are using it to justify other projects...