There are tradeoffs in both cases.

Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain.

You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though.

But, no AWS or AWS premium.

There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.

Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already.

Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...

The main value is that you can pay money to have decent network and storage with the hyperscalers. Which none of the intermediate-size cloud services offer to my knowledge?

> I could spend a few weeks (months?) compiling

why would you compile infra-stuff?! Usually this is nicely packaged...

> random GNU-certified "alternatives" for Cloud Services but

For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.

>Usually this is nicely packaged...

That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.

well, I don't know what you use? however

- ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too.

- Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon...

the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).

What kind of “services” do they provide that provide value in the kind of HPC setting you were talking about?

> trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...

A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish.

[1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior

It sounds like you're spending 7 figures of your research money without having done the most basic investigation of alternatives.

I spend low 6 figures of research money on hardware + staff each year, and this avoids us spending 7 figures on cloud costs + staff.

I guess having built the physical system I am aware of the tradeoffs... Though due to the funding structure and 0-cost colocation (for our unit), there was not a lot to be discussed and thus I'd be interested in actual numbers for comparison!