If it's not an MPI job, it's going to be painful.
What are you using for communication between nodes?
To make the tone clear I'm genuinely interested as I'm not sure what running on a Cray prevents you from doing?
We don't need to communicate between the nodes; We don't need MPI. We need distributed databases and we might need Spark. We need machine learning. We need servers. We would like availability and reliability.
We get full batch nodes that are relatively anemic in memory for our workload (4GB/core) for a maximum of 24 hours at a time.
Which just emphasizes the point that most applications would be better served by cheaper, more resilient hardware.
Huh. At the core of almost every problem I've ever encountered in my 20+ year career programming that I would consider "interesting" is the need to allocate work across many compute nodes that have some amount of shared state. You could just as well take A* as a baseline for these types of problems. Your typical cloud computing infra/design is not a good match for these.
It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state.
There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core).
So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.
Or you can send get the network admins to allow you to send them a email, but the first suggestion sounds more fun :P
1) I have a rail system that I want to optimize train movement through the topology. My search space is the number of running locomotives to the power of the number of switches in the system. On a continental rail system, that's a number with many, many, many zeros after it. To optimize this, I have many cores chewing through the space, but to know where to search they must share a visited node cache otherwise they will search each others space and waste effort.
2) I have a system that predicts part failures by collecting sensor data of the machine in a time series. I must look for patterns of change in these sensor readings and compare that to patterns that I know about, often drilling through data over many weeks looking for feature alignment in the sensor matrices. Same problem: many cores, some amount of space that is known vs. unknown that must be shared, but as a result is a contentious resource.
On the other hand, you can just run N independent simulations, and pool the statistics. In many cases, this is as good as if not better than running one long MD simulation. Here, all the processors share a common input file, but it's a single file, it only has to be communicated once. Any results from the processors are compacted, and sent to a large file server, which in itself is designed with partitioning since all the output files are independent. This approach won't let you work with large systems (unless your individual servers have more RAM), but it scales linearly to far more processors than tightly coupled parallelism, for far cheaper, and often, the scientific results are as good or better.
This shows up in many cases- in my 20+ year career programming, nearly all the problems I saw people run on supercomputers eventually could be run on CPUs with easier code and better performance. There are some irreducible codes for which only supercomputers are capable of meeting the required specs, and those are the only codes which should run on supercomputers. Otherwise you're pissing money on interconnect and hard to program environments).
In short: if you can convert your supercomputer job into a cloud cluster, you should do so, because it will save time and money. If you can't, then you should use a supercomputer.
Note also that cloud interconnects have gotten significantly better over the years- for me, 100Gbit throughput and latency (as long as the cluster has the total parallel bisection bandwidth) is fine for me and most other people.
/I work for Basho/
5 Preliminary progress of scientific computing applications on the TaihuLight
5.1 Refactoring the community atmospheric model (CAM) on the Sunway TaihuLight
5.2 A fully-implicit nonhydrostatic dynamic solver for cloud-resolving atmospheric simulation on eight million cores
5.3 A highly effective global surface wave numerical simulation with ultra-high resolution
5.4 Peta-scale atomistic simulation of silicon nanowires
5.5 Large-scale phase-field simulation for coarsening dynamics based on Cahn-Hilliard equation with degenerated mobility
5.6 Application: summary and comparison
What's the evidence for this claim? AFAIK the DoE INCITE awards -- which is how you get allocated time on the big supercomputers -- are quite competitive.
Also the supercomputers don't cost billions. Aurora, for example only costs $200 million, and if it lasts as long as its predecessor Mira did, that is $200 million spread over 6 years, which is a pretty small number.
> My conclusion was most of these computers aren't needed
Every dollar spent on building these computers is a dollar NOT spent in the budget of a research-driven grant. Give these dollars to physicists, mathematicians, genomics scientists, chemists, climate people etc. This way the demand in computing will dictate the type of systems built.
p.s. I've attended the event in person that's listed in the article hosted by the OSTP, which is a part of the White House (National Strategic Computing Initiative). My conclusion was most of these computers aren't needed and a misdirection of funds to the tune of billions of dollars that computing / scientific computing could really benefit from in other ways.