News for hackers. Stuff that matters.

A comment on AWS Multiple Services Down in us-east-1. Parent comment

Re:AWS Multiple Services Down in us-east-1 by rvba on Monday October 20, 2025 @02:43PM (#45647557)
In a company where I worked, the tool measuring downtime was at the same server, so even if the server was down they still showed 100% up.

If the server didnt work - the tool too measure didnt work too! Genius

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by bityard on Monday October 20, 2025 @02:58PM (#45647755)
This happened to AWS too.

February 28, 2017. S3 went down and took down a good portion of AWS and the Internet in general. For almost the entire time that it was down, the AWS status page showed green because the up/down metrics were hosted on... you guessed it... S3.

https://aws.amazon.com/message/41926/

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by subpar on Monday October 20, 2025 @04:19PM (#45648786)
obligatory https://x.com/lintzston/status/791761626890469377
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by belter on Monday October 20, 2025 @04:42PM (#45649077)
Published in the same week of October ...9 years ago ...Spooky...
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by hinkley on Monday October 20, 2025 @07:41PM (#45650804)
Five times is no longer a couple. You can use stronger words there.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by bapak on Tuesday October 21, 2025 @02:44AM (#45653118)
It happened a murder of times.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by hinkley on Tuesday October 21, 2025 @01:38PM (#45658830)
Ha! Shall I bookmark this for the eventual wiki page?
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by casey2 on Wednesday October 22, 2025 @06:44AM (#45667198)
https://www.youtube.com/watch?v=HxP4wi4DhA0

Maybe they should start using real software instead of mathematicians' toy langs

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by Scoundreller on Monday October 20, 2025 @06:02PM (#45649996)
Have we ever figured out what “red” means? I understand they’ve only ever gone to yellow.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by kokanee on Monday October 20, 2025 @07:03PM (#45650528)
If it goes red, we aren't alive to see it
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by Cthulhu_ on Tuesday October 21, 2025 @04:35AM (#45653723)
I'm sure we need to go to Blackwatch Plaid first.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by decimalenough on Monday October 20, 2025 @03:30PM (#45648141)
I used to work at a company where the SLA was measured as the percentage of successful requests on the server. If the load balancer (or DNS or anything else network) was dropping everything on the floor, you'd have no 500s and 100% SLA compliance.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by conductr on Monday October 20, 2025 @03:15PM (#45647965)
Similar to hosting your support ticketing system with same infra. "What problem? Nobody's complaining"
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by bigiain on Monday October 20, 2025 @06:11PM (#45650078)
I spent enough time ~15 years back to find an external monitoring service that did not run on AWS and looked like a sustainable business instead of a VC fueled acquisition target - for our belts-n-braces secondary monitoring tool since it's not smart to trust CloudWatch to be able to send notifications when it's AWS's shit that's down.

Sadly while I still use that tool a couple of jobs/companies later - I no longer recommend it because it migrated to AWS a few years back.

(For now, my out-of-AWS monitoring tool is a bunch of cron jobs running on a collections of various inexpensive vpses and my and other dev's home machines.)

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by 6031769 on Tuesday October 21, 2025 @06:28AM (#45654279)
Nagios is still a thing and you can host it wherever you like.
[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by bigiain on Tuesday October 21, 2025 @11:21PM (#45664604)
Interestingly, the reason I originally looked for and started using it was an unapproved "shadow IT" response to an in-house Nagios setup that was configured and managed so badly it had _way_ more downtime than any of the services I'd get shouted about at if customers noticed them down before we did...

(No disrespect to Nagios, I'm sure a competently managed installation is capable of being way better than what I had to put up with.)

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by hinkley on Monday October 20, 2025 @07:40PM (#45650791)
I’ve been customer for at least four separate products where this was true.

I can’t explain why Saucelabs was the most grating one, but it was. I think it’s because they routinely experienced 100% down for 1% of customers, and we were in that one percent about twice a year. <long string of swears omitted>

[ Parent | on HN ]
Re:AWS Multiple Services Down in us-east-1 by AbstractH24 on Tuesday October 21, 2025 @10:39AM (#45656404)
If its not on the dashboard, it didn't happen
[ Parent | on HN ]