That's the problem:

We don't need to communicate between the nodes; We don't need MPI. We need distributed databases and we might need Spark. We need machine learning. We need servers. We would like availability and reliability.

We get full batch nodes that are relatively anemic in memory for our workload (4GB/core) for a maximum of 24 hours at a time.

Basically it's hard to use a Cray node as a stand-alone server. Yes it runs linux, but it has no storage, no external network. There is the so-called cluster compatibility mode but bottom line everything is a scheduled batch job in a very walled garden.

Cray invests a lot in their interconnect; if you aren't using it, there is little reason to be on a Cray.

Which just emphasizes the point that most applications would be better served by cheaper, more resilient hardware.

All the stranger that they sold their interconnect technology, arguably their "secret sauce" to Intel. Without the interconnect, they are just a bunch of (very high quality) diskless x86 systems.

http://www.eetimes.com/document.asp?doc_id=1261622

> We don't need to communicate between the nodes

Huh. At the core of almost every problem I've ever encountered in my 20+ year career programming that I would consider "interesting" is the need to allocate work across many compute nodes that have some amount of shared state. You could just as well take A* as a baseline for these types of problems. Your typical cloud computing infra/design is not a good match for these.

Well I guess computing in astrophysics and HEP isn't that interesting then :)

It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state.

There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core).

So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.

Well this is where you talk to the network admins at your university and get them in on a top secret program, code name: "It Was China", to hack into all the detectable ps4's on campus, call all those students to a hall for a random lottery social for a couple of hours while you commandeer there equipment, for science of course.

Or you can send get the network admins to allow you to send them a email, but the first suggestion sounds more fun :P

That, sir, is hilariously awesome.

Well here are two examples.

1) I have a rail system that I want to optimize train movement through the topology. My search space is the number of running locomotives to the power of the number of switches in the system. On a continental rail system, that's a number with many, many, many zeros after it. To optimize this, I have many cores chewing through the space, but to know where to search they must share a visited node cache otherwise they will search each others space and waste effort.

2) I have a system that predicts part failures by collecting sensor data of the machine in a time series. I must look for patterns of change in these sensor readings and compare that to patterns that I know about, often drilling through data over many weeks looking for feature alignment in the sensor matrices. Same problem: many cores, some amount of space that is known vs. unknown that must be shared, but as a result is a contentious resource.

In many cases, problems partition nicely across machines without communication. Take molecular dynamics as an example of an "interesting" problem. One approach is MPI-based parallelism: partition the particles or other entities in the system across specific processors, compute all the individual forces, sum them up, and send the results to the processors that need it. This allows you a number of advantages: you can work with larger systems (ones that wouldn't fit in RAM on a single machine), you can speed up the total throughput, etc). But, it comes at a huge cost to developer productivity, and the machines have to have very low latency, high through primitives, so that all the processors are evenly balanced (Amdahl's law places upper bound on total throughput).

On the other hand, you can just run N independent simulations, and pool the statistics. In many cases, this is as good as if not better than running one long MD simulation. Here, all the processors share a common input file, but it's a single file, it only has to be communicated once. Any results from the processors are compacted, and sent to a large file server, which in itself is designed with partitioning since all the output files are independent. This approach won't let you work with large systems (unless your individual servers have more RAM), but it scales linearly to far more processors than tightly coupled parallelism, for far cheaper, and often, the scientific results are as good or better.

This shows up in many cases- in my 20+ year career programming, nearly all the problems I saw people run on supercomputers eventually could be run on CPUs with easier code and better performance. There are some irreducible codes for which only supercomputers are capable of meeting the required specs, and those are the only codes which should run on supercomputers. Otherwise you're pissing money on interconnect and hard to program environments).

In short: if you can convert your supercomputer job into a cloud cluster, you should do so, because it will save time and money. If you can't, then you should use a supercomputer.

Note also that cloud interconnects have gotten significantly better over the years- for me, 100Gbit throughput and latency (as long as the cluster has the total parallel bisection bandwidth) is fine for me and most other people.

Aha thanks! Yes I agree with you points here, not an ideal workload for a Cray system.

Talk to me about Riak.

/I work for Basho/