If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it.
Best thing about your stuff is that it's literally all funny precisely because it's all true.
Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast.
We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day.
Completely deadpan, and in front of several VPs of a forth one 50.
I turn to guy to my right who lived nearby. “Go home and get your chainsaw”
We were quickly let in. Also got fast approval to install proper cooling.
Fire extinguisher nearby, smart temp sensors, but still...
I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts.
Do you have any recommendations?
Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors.
Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses...
Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened
Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem
For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.
Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be.
Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.
Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting.
Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?
My PhD involved quite a lot of high performance computing simulations. We kept running into problems where on warm days, all our jobs would get killed pretty consistently. Our IT guy noticed a pattern where the temps would be perfectly normal, and then suddenly out of nowhere go through the roof, triggering a thermal shutdown of the racks.
In the end our IT guy camped out in the server room on a hot day to watch what was happening. The Astrophysics' cabinet were directly infront of ours, and they had jerry rigged their cabinet door so that when it got too hot, it would swing open and their hot air would be blown all over the neighbouring cabinets...
That said, many scientists are operating on premise hardware like this: some servers in a shared rack and an el-cheapo storage solutions with an ssh access for people working in the lab. And it works just fine for them.
Cloud services focus for running business computing in a cloud, emphasizing recurring revenue. Most research labs are much more comfortable with spending the hardware portion of a grant upfront and not worrying about some student who, instead of working on some fluid dynamics problem found a script to re-train a stable diffusion and left it running over winter break. My 2c.
Until it doesn't because there's a fire or huge power surge or whatever.
That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.
When your own hardware rack goes down. You know the problem, how much it costs to fix it, and when it will come back up; usually within a few hours (or minutes) of it going down.
Do things catch fire, yes. But I think you’re over-estimating how often. In my entire life, I’ve had a single SATA connector catch fire and it just melted plastic before going out.
With AWS it's extremely easy to keep an up-to-date database backup in a different region.
And it's great that you haven't personally encountered disaster, but of course once again that's cherry-picking. And it's not just a component overheating, it's the whole closet on fire, it's a broken ceiling sprinkler system going off, it's a hurricane, it's whatever.
For the rest, there’s insurance. Most calculations done in a research setting are dependent upon that research surviving. If there’s a fire and the whole building goes down, those calculations are probably worthless now too.
Hell, most companies probably can’t survive their own building/factory burning down.
It is just as extremely easy on Hetzner or on premises
For home gamers like myself, it's has become a no brainer with advances in tunneling, docker, and cheap prices on Ebay.
An outage, or even permanent loss of hardware, might not be a big problem if you're just running computations on data of which you have multiple copies. At worst, you might have to copy a few TB of data from an external hard drive and redo a few weeks' worth of computations.
I see it the other way: experimental scientists operate with unreliable systems all the time: fickle systems, soldered one-time setups, shared lab space, etc. Computing is just one more thing that is not 100% reliable (but way more reliable than some other equipment), and usb data sticks serve as a good enough data backup.
I'd further say that you're probably over-estimating how valuable mitigating that risk is to anyone, although there are a few limited set of customers that genuinely do care.
There are few places I can think of that would benefit more by avoiding cloud costs than scientific computing...
They often have limited budgets that are driven by grants, not derived by providing online services (computer going down does not impact bottom line).
They have real computation needs that mean hardware is unlikely to sit idle.
There is no compelling reason to "scale" in the way that a company might need to in order to handle additional unexpected load from customers or hit marketing campaigns.
Basically... the only meaningful offering from the cloud is likely preventing data loss, and this can be done fairly well with a simple backup strategy.
Again - they aren't a business where losing a few hours/days of customer data is potentially business ending.
---
And to be blunt - I can make the same risk avoidance claims about a lot of things that would simply get me laughed out of the room.
"The lead researcher shouldn't be allowed in a car because it might crash!"
"The lab work must be done in a bomb shelter in case of war or tornados!"
"No one on the team can eat red meat because it increases the risk of heart attack!"
and on and on and on... Simply saying "There's risk" is not sufficient - you must still make a compelling argument that the cost of avoiding that risk is justified, and you're not doing that.
Consider the following, I have never considered applying meltdown or spectre mitigations if it makes my code run slower because I plain don't care, assuming anyone even peeks at what my simulations doing, whoopdeedo, I don't care. I won't do that on my laptop I use to buy shit off amazon with, but the workstation I have control of? I don't care. I DO care if my simulation will take 10 days instead of a week.
My use case isn't yours because my needs aren't yours. Not everything maps across domains.
Some sort of 80/20 principle is at works here. Most of the costs in professional cloud solutions comes from making the infrastructure 99.99% reliable instead of 99% reliable. It is totally worth it if you have millions of customers that expect a certain level of reliability, but a complete overkill if the worst case scenario from a system failure is some graduate student having to redo a few days worth of computations (which probably had to be redone several times anyway because of some bug in the code or something).
I found this article: https://journals.sagepub.com/doi/10.1177/1073110520917030
Their budget for the network gear was a couple hundred bucks and some old garbage consumer grade network gear. For something that spit out 10s of GB a second (at least) across a ton of network connections (they didn't seem to know what would even happen when they ran it), and was so bursty all but the highest end of gear could handle it.
Can confirm sometimes scientists aren't really up on the overall costs. Then they dump it "this isn't working" on their university IT team to absorb the costs / manpower costs.
The article points out that this is mostly not necessary for scientific computing.
Web apps quickly become finely tuned factory machines, executing a million times a day and being duplicated thousands of times.
Scientific computing projects are often more like workshops. Getting charged by the second while you're sitting at a console trying to figure out what this giant blob you were sent even is is unpleasant. The solution you create is most likely to be run exactly once. If it is a big hit, it may be run a dozen times.
Trying to run scientific workloads on the cloud is like trying to put a human shoe on a horse. It might be possible but it's clearly not designed for that purpose.
Classic big workstations are way more capable than people think - but at the same time it's hard to justify buying one machine per user unless your department is swimming in money. Also, academic budgets tend to come in fixed chunks, and university IT departments may not have your particular group as a priority - so often it's just better to invest once in a standalone server tower that you can set up to do exactly what you need it to than try to get IT to support your needs or the accounting department to pay recurring AWS bills.
Properly provisioned linux machines need no maintenance. You drive them until there is a hardware failure.
At the other end of the spectrum there's labs [0] where the scientists need to carefully study and develop increasing skills at operating the electronics the way IT wants it done, and even worse of a distraction when there's a moving target keeping up with the IT electronics approach changing faster than the progress most labs make in their own scientific field.
What labs need more of is your kind of IT operator who can bring that option (your extreme end of the spectrum in favor of lab workers) within reach for when it is the most appropriate choice.
When labs fail to retain such adequate talent, they can rule out that option going forward, and that's one less tool in the toolbox.
[0] Including many which have good records of breakthrough progress before becoming computerized to begin with.
Running a modern AMD-based server that has 48 cores, at least 192 GB of RAM, and no included disk space costs:
~$2670.36/mo for a c5a.24xlarge AWS on-demand instance
~$1014.7/mo for a c5a.24xlarge AWS reserved instance on a three-year term, paid upfront
~$558.65/mo on OVH Cloud[1]
~$512.92/mo on Hetzner[2]
~$200/mo on your own infrastructure as a large institution[3]
"Assumes an AMD EPYC 7552 run at 100% load in Boston with high electricity prices of $0.23/kWh, for $33.24/mo in raw power. Hardware is amortized over five years, for an average monthly price of $67.08/mo. We assume that your large institution already has 24/7 security and public internet bandwidth, but multiply base hardware and power costs by 2x to account for other hardware, cooling, physical space, and a half-a-$120k-sysadmin amortized across 100 servers."
For a large university it probably makes sense to have and manage their own compute infrastructure (cheap post-doc labor, ftw!) but for smaller outfits, AWS can make a lot of sense for scientific computing (said as someone who uses AWS for scientific computing), especially if you have fluctuating loads.
What works best IMO (and what we do) is have a minimum-to-moderate amount of compute resources in house that can satisfy the processing jobs most commonly run (and where you haven't had to overinvest in hardware), and then switch to AWS/other for heavier loads that run for a finite period.
Another problem with in-house hardware is that you spent all that money on Nvidia V100's a few years ago and now there's the A100 that blows it away, but you can't just switch and take advantage of it without another huge capital investment.
Power (especially if there is some kind of significant scientific facility on premise), space (especially in reused buildings), manpower (undergrads, grad students, post docs, professional post graduates), running old/reused hardware, etc...
You can get away with those at large research universities. Some of that you can get away with at national lab sorts of places (not going to find as much free/cheap labor, surplus hardware). If you start going down in scale/prestige, etc... none of that holds true.
Running a bunch of hardware from the surplus store in a closet somewhere with Lasko fans taped to the door is cheap. To some extent, the university system encourages such subsidies.
In any case, once you get to actually building a datacenter, if you have to factor in power, if you have a 4 hardware refresh cycle, professional staffing, etc... unless you are in one of those low CoL college towns - cloud is probably more more than 1.5 to 3x more expensive for compute (spot, etc...). Storage on prem is much cheaper - erasure coded storage systems are cheap to buy and run, and everybody wants their own high performance file system.
One continuing cloud obstacle though - researchers don't want to spend their time figuring out how to get their code friendly to preemptible VMs - which is the cost effective way to run on cloud.
Another real issue with sticking to on-prem HPC is talent acquisition and staff development. When you don't care about those things so much, it's easy to say it's cheap to run on-prem, but often the pay is crap for the required expertise, and ignoring cloud doesn't help your staff either.
Less than a year later a story made national headlines that a professor with direct access to classified material had mysteriously disappeared & never disclosed his close ties with their home country. So not only should you be worried about back doors and side doors; but also watch what's going through your front door as well!
All that being said, I have never worked at a place (or in a dept) whose threat profile made APT a real thing.
You're fortunate if APTs don't consider you a worthy target. They are no joke, and in most cases are playing a long game, more interested in penetration, persistent presence, and quiet theft of information, than in doing anything you'd notice - until they aren't. We had one who burned an asset that had been cultivated on our network for a couple of years to make a hard press play for information about a very particular high profile patient immediately after that patient had been seen (the fact that the patient was seen was public information). But with over 20,000 servers and 300,000 total nodes on the network - some of which cannot be fully patched because they run software someone has to have access to but which won't run on the newest versions of OSs or databases - you still don't know what else they've burrowed into.
A big tech company supplier to our organization had a very telling incident that ended up being detected on our side of the connection, where a brief mistake by network admins opened a channel through their layers of protection for their buid pipeline. In the minutes it was open, an APT detected the access (likely because they already owned something internal), and inserted code into their certified OS build pipeline, which we ended up with in our institution.
Externally-hosted HPC means every single compute job is seen as something that directly costs money. This negatively affects the quality of scientific output (research playfulness / creativity / focus on the research / etc).
Seen by whom? Dummies it seems. Maybe the dummies are the problem, not the computing or accounting models.
As far as HPC goes specifically, we could get some of the financial numbers to make sense in the cloud (cpu intensive jobs), but couldn't make it work for others (data intensive jobs shipping PB's around on the reg)
That and HPC has a lot of grant funding. It can be quite advantageous for an org to use an on prem data center almost like a slush fund. Can keep key projects running that would otherwise die when they are having a rough funding year.
Put like for like in a well managed data center against negotiated and planned cloud services, and the former may still win, but it won't be dramatically cheaper, and figured over depreciable lifetime and including opportunity cost, may cost more. It takes work to figure out which is true.