I think it's a W, so far.
Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain.
You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though.
But, no AWS or AWS premium.
Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
> I could spend a few weeks (months?) compiling
why would you compile infra-stuff?! Usually this is nicely packaged...
> random GNU-certified "alternatives" for Cloud Services but
For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.
That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.
- ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too.
- Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon...
the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).
> trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish.
[1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior
I spend low 6 figures of research money on hardware + staff each year, and this avoids us spending 7 figures on cloud costs + staff.
I should have really mentioned your(our) use case, as smallish department without dedicated access to a cluster or people who can run a real steel cluster.
AWS's one click lustre clustre service (FSx) combined with Batch is a really quick and powerful basic HPC cluster.
Before we got bought out, we had planned for creating a "base load" of real steel machines to process the continuous jobs, and spin up the rest of the cluster as and when demand needed
Cloud is cheaper for such a workload, yes. But you still wouldn't want to pay the AWS premium for that.
But I guess nobody ever got fired for choosing AWS.
If you want high speed storage, thats reliable and zero cost to setup (ie doesn't require staff to do) then AWS and spot instances is very much a cheap option.
If you have in house expertise and can knock up a reliable high speed storage cluster (no porting to s3 is not the right option, that takes years and means you can't edit files inline.) then running on one of the other cloud providers is an option.
But they often can't cope with the scale, or don't provide the right support.
Ah, the fallacy of the cloud not requiring staff to magically work. Funnily enough, every time I heard this IRL, it was coming from dudes who was in fact paid to manage AWS instances and services for their company as their day job but somehow never included his salary and the one of his colleagues.
> But they often can't cope with the scale
Most people vastly overestimate the scale they need or underestimate the scale of even second tier cloud providers but there's not many workload that could cause storage trouble anywhere. For the record a single rack can host 5/10 PB, how many people needs multiple exabytes of storage again?
> or don't provide the right support.
I've never been a whole-data-centers-scale customer but my experience with AWS support didn't leave me in awe.
its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking.
Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.)
As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter.
> Most people vastly overestimate the scale they need
Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.)
Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations.
> AWS support didn't leave me in awe.
Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.
The fallacy is that it costs zero staff. But in reality it doesn't, it reduces the amount of staff you need, but only to an extend, because you still need plenty of people to run and monitor your cloud infra.
> Yes, but we are talking about an HPC cluster that is >100k nodes.
It is also much higher requirements than anything AWS is used to run, so you either have the know-how in house or don't have it at all.
> Second tier cloud providers are rarely able to provide 10k GPUs on the spot,
If you need 10k GPUs on the spot, then you have a serious planning issue… I know that many companies have incompetent management, but that's not the kind of thing you're supposed to brag about in an internet discussion. And using AWS to cope with management and planning deficiency is only going to get you this far, at some point you'll need to address them eventually.
Yes, and no. As a replacement for a reasonably well used cluster, its going to be super expensive. Something like 3-5x the price.
For providing burst capacity, its significantly cheaper/faster, and can be billed directly.
But would I recommend running your stuff directly on AWS for compute heavy batch processing? no. if you are thinking about hiring machines in a datacenter, then yes.