When your own hardware rack goes down. You know the problem, how much it costs to fix it, and when it will come back up; usually within a few hours (or minutes) of it going down.
Do things catch fire, yes. But I think you’re over-estimating how often. In my entire life, I’ve had a single SATA connector catch fire and it just melted plastic before going out.
With AWS it's extremely easy to keep an up-to-date database backup in a different region.
And it's great that you haven't personally encountered disaster, but of course once again that's cherry-picking. And it's not just a component overheating, it's the whole closet on fire, it's a broken ceiling sprinkler system going off, it's a hurricane, it's whatever.
For the rest, there’s insurance. Most calculations done in a research setting are dependent upon that research surviving. If there’s a fire and the whole building goes down, those calculations are probably worthless now too.
Hell, most companies probably can’t survive their own building/factory burning down.
It is just as extremely easy on Hetzner or on premises
For home gamers like myself, it's has become a no brainer with advances in tunneling, docker, and cheap prices on Ebay.
An outage, or even permanent loss of hardware, might not be a big problem if you're just running computations on data of which you have multiple copies. At worst, you might have to copy a few TB of data from an external hard drive and redo a few weeks' worth of computations.
I see it the other way: experimental scientists operate with unreliable systems all the time: fickle systems, soldered one-time setups, shared lab space, etc. Computing is just one more thing that is not 100% reliable (but way more reliable than some other equipment), and usb data sticks serve as a good enough data backup.
I'd further say that you're probably over-estimating how valuable mitigating that risk is to anyone, although there are a few limited set of customers that genuinely do care.
There are few places I can think of that would benefit more by avoiding cloud costs than scientific computing...
They often have limited budgets that are driven by grants, not derived by providing online services (computer going down does not impact bottom line).
They have real computation needs that mean hardware is unlikely to sit idle.
There is no compelling reason to "scale" in the way that a company might need to in order to handle additional unexpected load from customers or hit marketing campaigns.
Basically... the only meaningful offering from the cloud is likely preventing data loss, and this can be done fairly well with a simple backup strategy.
Again - they aren't a business where losing a few hours/days of customer data is potentially business ending.
---
And to be blunt - I can make the same risk avoidance claims about a lot of things that would simply get me laughed out of the room.
"The lead researcher shouldn't be allowed in a car because it might crash!"
"The lab work must be done in a bomb shelter in case of war or tornados!"
"No one on the team can eat red meat because it increases the risk of heart attack!"
and on and on and on... Simply saying "There's risk" is not sufficient - you must still make a compelling argument that the cost of avoiding that risk is justified, and you're not doing that.
Consider the following, I have never considered applying meltdown or spectre mitigations if it makes my code run slower because I plain don't care, assuming anyone even peeks at what my simulations doing, whoopdeedo, I don't care. I won't do that on my laptop I use to buy shit off amazon with, but the workstation I have control of? I don't care. I DO care if my simulation will take 10 days instead of a week.
My use case isn't yours because my needs aren't yours. Not everything maps across domains.
Some sort of 80/20 principle is at works here. Most of the costs in professional cloud solutions comes from making the infrastructure 99.99% reliable instead of 99% reliable. It is totally worth it if you have millions of customers that expect a certain level of reliability, but a complete overkill if the worst case scenario from a system failure is some graduate student having to redo a few days worth of computations (which probably had to be redone several times anyway because of some bug in the code or something).
Until it doesn't because there's a fire or huge power surge or whatever.
That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.