AWS doesn't make sense for scientific computing

Re:AWS doesn't make sense for scientific computing by W-Stool on Friday October 07, 2022 @12:43PM (#33123744)

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. The closet with racks of gear and no cooling is another fond memory. Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by 0xbadcafebee on Friday October 07, 2022 @12:48PM (#33123817)

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by eastbound on Friday October 07, 2022 @01:15PM (#33124215)

Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by bilbo0s on Friday October 07, 2022 @01:55PM (#33124741)

A bit off topic, but I gotta say you guys are a riot!

If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it.

Best thing about your stuff is that it's literally all funny precisely because it's all true.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by geoduck14 on Friday October 07, 2022 @08:40PM (#33128725)

I don't know why you are being down voted. I'd watch a Netflix special on this stuff

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by treeman79 on Friday October 07, 2022 @06:20PM (#33127553)

We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often.

Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast.

We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day.

Completely deadpan, and in front of several VPs of a forth one 50.

I turn to guy to my right who lived nearby. “Go home and get your chainsaw”

We were quickly let in. Also got fast approval to install proper cooling.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by pbronez on Friday October 07, 2022 @05:07PM (#33126862)

This is my fear about my homelab lol

Fire extinguisher nearby, smart temp sensors, but still...

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by rovr138 on Friday October 07, 2022 @05:54PM (#33127342)

oh, nice idea with temp sensor.

I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts.

Do you have any recommendations?

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by W-Stool on Friday October 07, 2022 @05:56PM (#33127362)

What are you using for a homelab priced temperature sensor?

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by xani_ on Friday October 07, 2022 @06:57PM (#33127952)

Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp.

Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors.

Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by xani_ on Friday October 07, 2022 @07:03PM (#33128021)

No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works.

Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses...

Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened

Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem

For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by absolutelynobo on Friday October 07, 2022 @08:35PM (#33128694)

Cool how you use the acronym RCD and don’t expand it anywhere. Or any of the other homelab acronyms you used!

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by tiagod on Saturday October 08, 2022 @12:44AM (#33129829)

RCD is not a homelab acronym, its a type of circuit breaker.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by halper on Saturday October 08, 2022 @12:43PM (#33133804)

"Residual current device". It detects current leaking to ground; essentially it prevents you from killing someone by throwing a toaster into the bathtub.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by walnutclosefarm on Friday October 07, 2022 @09:18PM (#33128926)

I never saw an actual fire, but I we did see smoke, in a closet, on a floor that was 90% patient care, but happened to have a research area as well because the research needed access to expensive radiography equipment. The close was literally stuffed with what amounted to gaming machines, purchased with grant money through an importer, directly from China. The guy who set them up was smart enough to put them all behind a little firewall, so the enterprise network couldn't see them. It was a (literally) hot mess, both infrastructure and security-wise.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by bluedino on Friday October 07, 2022 @02:54PM (#33125435)

It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by davidmr on Friday October 07, 2022 @05:29PM (#33127095)

That was definitely one of the weirdest things of working in academia IT: “hey. Can you buy me a workstation that’s as close to $6,328.45 as it is possible to get, and can you do it by 4pm?”

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by kjs3 on Monday October 10, 2022 @12:53PM (#33153707)

Same thing happens in the government sector here (US). If you don't spend all of the budget you requested last year, you might not get it next year. There is an entire ecosystem of bottom-feeder GSA companies that apparently exist to spend year-end money that would otherwise go to 'waste'.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by systemvoltage on Friday October 07, 2022 @05:00PM (#33126795)

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data."

I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by capableweb on Friday October 07, 2022 @05:45PM (#33127253)

> think AWS is some sort of a massively expensive enterprise thing

Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be.

Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by pwinnski on Friday October 07, 2022 @07:04PM (#33128034)

At $15/month, it would take longer than a typical school year to beat.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by gonzo41 on Friday October 07, 2022 @06:15PM (#33127519)

Don't say the budget outloud near AWS. They'll find a way to help you spend it.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by systemvoltage on Friday October 07, 2022 @06:31PM (#33127653)

Hahaha, may be I need to just go into the AWS ether and start yakking big words like "Elastic Kubernetes Service" to confuse the scientists and get my aws fix. These people are too stingy. I want some shit running in AWS, what good is this admin IAM role.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by absolutelynobo on Friday October 07, 2022 @08:36PM (#33128701)

GP has already lost that battle.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by datavirtue on Saturday October 08, 2022 @08:35AM (#33131781)

Yeah, by following the "documentation" and "best practices."

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by geoduck14 on Friday October 07, 2022 @08:43PM (#33128737)

We use a phrase around my job: the only attack we suffer from is "denial of wallet"

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by slavik81 on Saturday October 08, 2022 @02:20AM (#33130231)

What are you getting for $15/mo? That's surprisingly inexpensive for scientific compute.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by funstuff007 on Friday October 07, 2022 @07:28PM (#33128227)

I cannot wait for Ampere servers to become widespread. Most of the power and cooling issues will go away.

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by getcrunk on Saturday October 08, 2022 @06:09PM (#33136509)

sarcasm?

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by absolutelynobo on Friday October 07, 2022 @08:33PM (#33128684)

> Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store.

Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting.

Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?

[ Parent | on HN ]

Re:AWS doesn't make sense for scientific computing by morbia on Saturday October 08, 2022 @06:19AM (#33131170)

Your comment about cooling reminded me of a fun anecdote from my time in academia.

My PhD involved quite a lot of high performance computing simulations. We kept running into problems where on warm days, all our jobs would get killed pretty consistently. Our IT guy noticed a pattern where the temps would be perfectly normal, and then suddenly out of nowhere go through the roof, triggering a thermal shutdown of the racks.

In the end our IT guy camped out in the server room on a hot day to watch what was happening. The Astrophysics' cabinet were directly infront of ours, and they had jerry rigged their cabinet door so that when it got too hot, it would swing open and their hot air would be blown all over the neighbouring cabinets...

[ Parent | on HN ]