Speaking of Beowulf, has there ever been an evolution of the concept? The closest I've seen since has been QNX's QNet, which allows transparent management and communication between process on nodes of the cluster. I suppose Hadoop or even Kubernetes can be seen as the continuation of the concept?
In some ways this idea came to dominate. If you look at the top500 list:
https://www.top500.org/list/2016/11/
All of these machines are big clusters running Linux. Mostly on Intel CPUs.
But on the other hand, the idea of using commodity hardware is kind of a thing of the past. It's mostly Xeon CPUs, not desktop processors. And it's specialized network hardware. And more and more you see dedicated compute hardware like Intel Phi and Nvidia Tesla cards.
And thinking of he main network room, with the amount of brocades in there, it's probably more expensive than the main enterprise pod just in sheer super-expensive network stuffs.
We're also behind the times in lots of our management. 80% of our servers are bare metal, with limited automation. But we also do "NOC in a box"... many of our use cases wouldn't cleanly work right using tech like docker and kubernetes.
If you look through the archives of the beowulf mailing list, occasionally someone makes the argument you're making, and few people agree with it.
The rest of the non-phi/non-tesla hardware is pretty much off the shelf, but the interconnect is one of the two distinguishing features of a supercomputing-class cluster; the other is high-performance shared storage (which of course requires the interconnect to function).
https://twitter.com/hashcat/status/817367152927866880
and now to do machine learning...
I don't know if you'd call it an "evolution of the concept", but there are people who've made "low cost" clusters of Raspberry Pi boards (anywhere from four, to several hundred), not so much for practical purposes, but more for learning how to set up, use, and maintain such a system, without needing the space or power requirements a real system would need.
But, one fun thing I could imagine doing is using it as an incredibly portable PirateBox. Or any other use of a file server hiding in plain sight.