News for hackers. Stuff that matters.
Re:Microsoft Azure Outage by Uehreka on Wednesday October 29, 2025 @12:43PM (#45749410)
I noticed that Starbucks mobile ordering was down and thought “welp, I guess I’ll order a bagel and coffee on Grubhub”, then GrubHub was down. My next stop was HN to find the common denominator, and y’all did not disappoint.
Re:Microsoft Azure Outage by pants2 on Wednesday October 29, 2025 @01:01PM (#45749715)
Good thing HN is hosted on a couple servers in a basement. Much more reliable than cloud, it seems!
Re:Microsoft Azure Outage by dang on Wednesday October 29, 2025 @01:57PM (#45750608)
Re:Microsoft Azure Outage by hinkley on Wednesday October 29, 2025 @04:11PM (#45752402)
I’ve seen this up close twice and I’m surprised it’s only twice. Between March and September one year, 6 people on one team had to get new hard drives in their thinkpads and rebuild their systems. All from the same PO but doled out over the course of a project rampup. That was the first project where the onboarding docs were really really good, since we got a lot of practice in a short period of time.

Long before that, the first raid array anyone set up for my (teams’) usage, arrived from Sun with 2 dead drives out of 10. They RMA’d us 2 more drives and one of those was also DOA. That was a couple years after Sun stopped burning in hardware for cost savings, which maybe wasn’t that much of a savings all things considered.

Re:Microsoft Azure Outage by gogusrl on Wednesday October 29, 2025 @03:41PM (#45752014)
I got burnt by this bug on freakin' Christmas Eve 2020 ( https://forum.hddguru.com/viewtopic.php?f=10&t=40766 ). There was some data loss and a lot of lessons learned.
Re:Microsoft Azure Outage by airstrike on Wednesday October 29, 2025 @01:59PM (#45750648)
I love that "Ask HN: What'd you do while HN was down?" was a thing
Re:Microsoft Azure Outage by Cthulhu_ on Thursday October 30, 2025 @11:03AM (#45760830)
My plan B was going to the Stack Exchange homepage for some interesting threads but it got repetitive.
Re:Microsoft Azure Outage by praccu on Wednesday October 29, 2025 @10:27PM (#45755776)
Many years ago (13?), I was around when Amazon moved SABLE from RAM to SSDs. A whole rack came from a single batch, and something like 128 disks went out at once.

I was an intern but everyone seemed very stressed.

Re:Microsoft Azure Outage by Cthulhu_ on Thursday October 30, 2025 @11:02AM (#45760814)
Man I hit something like that once, a SSD had a firmware bug where it would stop working at an exact number of hours.
Re:Microsoft Azure Outage by lysace on Wednesday October 29, 2025 @01:17PM (#45749980)
It was on AWS at least (for a while) in 2022.

https://news.ycombinator.com/item?id=32030400

Re:Microsoft Azure Outage by jjice on Wednesday October 29, 2025 @02:01PM (#45750668)
Yeah looks like they're back on M5.

dang saying it's temporary: https://news.ycombinator.com/item?id=32031136

    $ dig news.ycombinator.com

    ; <<>> DiG 9.10.6 <<>> news.ycombinator.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54819
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;news.ycombinator.com.  IN A

    ;; ANSWER SECTION:
    news.ycombinator.com. 1 IN A 209.216.230.207

    ;; Query time: 79 msec
    ;; SERVER: 100.100.100.100#53(100.100.100.100)
    ;; WHEN: Wed Oct 29 13:59:29 EDT 2025
    ;; MSG SIZE  rcvd: 65
And that IP says it's with M5 again.
Re:Microsoft Azure Outage by parliament32 on Wednesday October 29, 2025 @01:07PM (#45749813)
Always has been.
Re:Microsoft Azure Outage by port11 on Wednesday October 29, 2025 @04:09PM (#45752384)
I noticed it when my Netatmo rigamajig stopped notifying me of bad indoor air quality. Lovely. Why does it need to go through the cloud if the data is right there in the home network…
Re:Microsoft Azure Outage by pasc1878 on Thursday October 30, 2025 @10:42AM (#45760560)
Same here for netatmo - ironically I replied to an incident report with netatmo saying all was OK when the whole system was falling over.

However netatmo does need to have a server to store data as you need to consolidate acreoss devices plus you can query gfor a year's data and that won't and can't be held locally.

Re:Microsoft Azure Outage by port11 on Friday October 31, 2025 @08:21AM (#45771235)
It could be local-first. I don't mind the cross-device sync being done centrally, of course, but the app specifically asks for access to Home and Local Network. I wonder if Home Assistant could deal with blackouts…
Re:Microsoft Azure Outage by hypeatei on Wednesday October 29, 2025 @12:57PM (#45749639)
Starbucks mobile was down during the AWS outage too...
Re:Microsoft Azure Outage by SoftTalker on Wednesday October 29, 2025 @01:00PM (#45749688)
They are multi-cloud --- vulnerable to all outages!
Re:Microsoft Azure Outage by mring33621 on Wednesday October 29, 2025 @01:13PM (#45749903)
you wouldn't believe some of the crap enterprise bigco mgmt put in place for disaster recovery.

they think that they are 'eliminating a single point of failure', but in reality, they end up adding multiple, complicated points of mostly failure.

Re:Microsoft Azure Outage by andoma on Wednesday October 29, 2025 @12:59PM (#45749680)
Go multi-cloud they said...
Re:Microsoft Azure Outage by Hamuko on Wednesday October 29, 2025 @12:58PM (#45749659)
Gonna build my application to be multicloud so that it requires multiple cloud platforms to be online at the same time. The RAID 0 of cloud computing.
Re:Microsoft Azure Outage by sergiotapia on Wednesday October 29, 2025 @01:05PM (#45749773)
Wow I just left a Starbucks drivethru line because it was just not moving. I guess it was because of this.
Re:Microsoft Azure Outage by iso1631 on Wednesday October 29, 2025 @03:40PM (#45752002)
You'd think that Starbucks execs would be held accountable for the fragile system they have put in place.

But they won't be.

Re:Microsoft Azure Outage by peanut-walrus on Wednesday October 29, 2025 @05:00PM (#45752957)
Why? Starbucks is not providing a critical service. Spending less money and resources and just accepting the risk that occasionally you won't be able to sell coffee for a few hours is a completely valid decision from both management and engineering pov.
Re:Microsoft Azure Outage by bobro on Thursday October 30, 2025 @01:20AM (#45756675)
Or maybe we should throw them in jail.
Re:Microsoft Azure Outage by DaSHacka on Thursday October 30, 2025 @04:15AM (#45757547)
I agree, but because the coffee is crap
Re:Microsoft Azure Outage by munchlax on Thursday October 30, 2025 @04:19AM (#45757573)
And ridiculously expensive
Re:Microsoft Azure Outage by bombcar on Thursday October 30, 2025 @05:25AM (#45757957)
It's absolutely batshit that an in-person transaction with cash becomes impossible when the computers are down.

I've seen it multiple times at various stores; only once did I see them taking cash and writing things down (probably to enter into the system later when it came back up).

Re:Microsoft Azure Outage by iso1631 on Thursday October 30, 2025 @06:37AM (#45758409)
If I were a Starbucks shareholder I wouldn't be happy that my company is throwing away revenue because of the CTO's decision to outsource accountability

Time and time again it's shown that AWS is far more expensive than other solutions, just easier for the Execs to offshore the blame.

Re:Microsoft Azure Outage by 01284a7e on Wednesday October 29, 2025 @02:27PM (#45751017)
Ha, maybe rethink the I AM NOTHING BUT A HUGE CLOUD CONSUMER thing on some fundamental levels? Like food?
Re:Microsoft Azure Outage by Theodores on Wednesday October 29, 2025 @03:36PM (#45751943)
My inner Nelson-from-the-Simpsons wishes I was on your team today, able to flaunt my flask of tea and homemade packed sandwiches. I would tease you by saying 'ha ha!' as your efforts to order coffee with IP packets failed.

I always go everywhere adequately prepared for beverages and food. Thanks to your comment, I have a new reason to do so. Take out coffees are actually far from guaranteed. Payment systems could go down, my bank account could be hacked or maybe the coffee shop could be randomly closed. Heck, I might even have an accident crossing the road. Anything could happen. Hence, my humble flask might not have the top beverage in it but at least it works.

We all design systems with redundancy, backups and whatnot, but few of us apply this thinking to our food and drink. Maybe get a kettle for the office and a backup kettle, in case the first one fails?

Re:Microsoft Azure Outage by jeffrallen on Wednesday October 29, 2025 @04:24PM (#45752562)
You know you can talk to your barista and ask for a bagel, right? If you're lucky they still take cash... if you still _have_ cash. :)
Re:Microsoft Azure Outage by 0_____0 on Wednesday October 29, 2025 @10:23PM (#45755754)
I was at a McDs a couple months back and I'm pretty sure you had to use the kiosk to order. Some places are deprecating the cashier entirely.
Re:Microsoft Azure Outage by Havoc on Wednesday October 29, 2025 @05:24PM (#45753252)
The sysadmin subreddit tends to beat hn on outage reports by an hour+ in my experience.

Bunch of on-call peeps over there that definitely know the instant something major goes down

Re:Microsoft Azure Outage by garbagewoman on Thursday October 30, 2025 @03:14AM (#45757220)
Service culture is so hollow