News for hackers. Stuff that matters.
Re:Tell HN: AWS us-east-1 services are down by Aachen on Monday October 20, 2025 @03:30AM (#45640875)
Signal is down from several vantage points and accounts in Europe, I'd guess because of this dependence on Amazon overseas

We're having fun figuring out how to communicate amongst colleagues now! It's when it's gone when you realise your dependence

Re:Tell HN: AWS us-east-1 services are down by tcumulus on Monday October 20, 2025 @03:55AM (#45641020)
Well, at least I now know that my Belgian university's Blackboard environment is running on AWS :)
Re:Tell HN: AWS us-east-1 services are down by Traubenfuchs on Monday October 20, 2025 @04:16AM (#45641186)
> We're having fun figuring out how to communicate amongst colleagues now!

When Slack was down we used... google... google mail? chat. When you go to gmail there is actually a chat app on the left.

Re:Tell HN: AWS us-east-1 services are down by BoredPositron on Monday October 20, 2025 @03:43AM (#45640936)
Our last resort fallbacks are channels on different IRC servers. They always hold.
Re:Tell HN: AWS us-east-1 services are down by Aachen on Monday October 20, 2025 @03:53AM (#45641001)
Self hosting is golden. Sadly we already feel like we have too many services for our company's size, and the sensitivity of vulnerabilities in customer systems precludes unencrypted comms. IRC+TLS could be used but we also regularly send screenshots and such in self-destructing messages (not that an attacker couldn't disable that, but to avoid there being a giant archive when we do have some sort of compromise), so we'd rather fall back to something with a similar featureset

As a degraded-state fallback, email is what we're using now (we have our clients configured to encrypt with PGP by default, we use it for any internal email and also when the customer has PGP so everyone knows how to use that)

Re:Tell HN: AWS us-east-1 services are down by bananapub on Monday October 20, 2025 @04:15AM (#45641180)
self-hosting isn't "golden", if you are serious about the reliability of complex systems, you can't afford to have your own outages impede your own engineers from fixing them.

if you seriously have no external low dep fallback, please at least document this fact now for the Big Postmortem.

Re:Tell HN: AWS us-east-1 services are down by Aachen on Monday October 20, 2025 @04:32AM (#45641304)
The engineers can walk up to the system and do whatever they need to fix them. At least, that's how we self host in the office. If your organisation hosts it far away then yeah, it's not self hosted but remote hosted
Re:Tell HN: AWS us-east-1 services are down by lmm on Monday October 20, 2025 @04:38AM (#45641349)
> The engineers can walk up to the system and do whatever they need to fix them.

Including fabricating new RAM?

Re:Tell HN: AWS us-east-1 services are down by Aachen on Monday October 20, 2025 @05:11AM (#45641665)
Including falling back to third-party hosting when relevant. One doesn't exclude the other

My experience with self hosting has been that, at least when you keep the services independent, downtime is not more common than in hosted environments, and you always know what's going on. Customising solutions, or workarounds in case of trouble, is a benefit you don't get when the service provider is significantly bigger than you are. It has pros and cons and also depends on the product (e.g. email delivery is harder than Mattermost message delivery, or if you need a certain service only once a year or so) but if you have the personell capacity and a continuous need, I find hosting things oneself to be the best solution in general

Re:Tell HN: AWS us-east-1 services are down by Nextgrid on Monday October 20, 2025 @06:01AM (#45642077)
Including fallback to your laptop if nothing else works. I saved a demo once by just running the whole thing from my computer when the Kubernetes guys couldn't figure out why the deployed version was 403'ing. Just had to poke the touchpad every so often so it didn't go to sleep.
Re:Tell HN: AWS us-east-1 services are down by filoleg on Monday October 20, 2025 @10:13AM (#45644148)
> Just had to poke the touchpad every so often so it didn't go to sleep

Unwarranted tip: next time, if you use macOS, just open the terminal and run `caffeinate -imdsu`.

I assume Linux/Windows have something similar built-in (and if not built-in, something that's easily available). For Windows, I know that PowerToys suite of nifty tools (officially provided by Microsoft) has Awake util, but that's just one of many similar options.

Re:Tell HN: AWS us-east-1 services are down by BobaFloutist on Monday October 20, 2025 @01:01PM (#45646255)
You can just turn of automatic sleep/screen off in Windows native power settings.
Re:Tell HN: AWS us-east-1 services are down by belmer2 on Monday October 20, 2025 @05:20AM (#45641751)
The key thing that AWS provides is the capacity for infinite redundancy. Everyone that is down because us-east-1 is down didn't learn the lesson of redundancy.
Re:Tell HN: AWS us-east-1 services are down by sgarland on Monday October 20, 2025 @07:00AM (#45642505)
Active-active RDBMS - which is really the only feasible way to do HA, unless you can tolerate losing consistency (or the latency hit of running a multi-region PC/EC system) - is significantly more difficult to reason about, and to manage.

Except Google Spanner, I’m told, but AWS doesn’t have an answer for that yet AFAIK.

Re:Tell HN: AWS us-east-1 services are down by senderista on Monday October 20, 2025 @11:36AM (#45645144)
Re:Tell HN: AWS us-east-1 services are down by yourapostasy on Monday October 20, 2025 @06:38AM (#45642347)
Some organizations’ leadership takes one look at the cost of redundancy and backs away. Paying for redundant resources most organizations can stomach. The network traffic charges are what push many over the edge of “do not buy”.

The cost of re-designing and re-implementing applications to synchronize data shipping to remote regions and only spinning up remote region resources as needed is even larger for these organizations.

And this is how we end up with these massive cloud footprints not much different than running fleets of VM’s. Just about the most expensive way to use the cloud hyperscalers.

Most non-tech industry organizations cannot face the brutal reality that properly, really leveraging hyperscalers involves a period of time often counted in decades for Fortune-scale footprints where they’re spending 3-5 times on selected areas more than peers doing those areas in the old ways to migrate to mostly spot instance-resident, scale-to-zero elastic, containerized services with excellent developer and operational troubleshooting ergonomics.

Re:Tell HN: AWS us-east-1 services are down by dd_xplore on Monday October 20, 2025 @05:20AM (#45641753)
If you self host, you must keep the spares, atleast for an enterprise environment.
Re:Tell HN: AWS us-east-1 services are down by worik on Monday October 20, 2025 @03:18PM (#45648006)
> in self-destructing messages (not that an attacker couldn't disable that, but to avoid there being a giant archive when we do have some sort of compromise

Admitting to that here?

In civilised jurisdictions that should be criminal.

Using cryptography to avoid accountability is wrong. Drug dealing and sex work, OK, but in other businesses? Sounds very crooked to me

Re:Tell HN: AWS us-east-1 services are down by jamwil on Monday October 20, 2025 @04:18PM (#45648764)
They're not sidestepping accountability, or at least we can't infer that from the information volunteered. They're talking about retention, and the less data you retain (subject to local laws), the less data can be leaked in a breach.
Re:Tell HN: AWS us-east-1 services are down by endre on Monday October 20, 2025 @03:47AM (#45640965)
same here, we never left IRC
Re:Tell HN: AWS us-east-1 services are down by lexandstuff on Monday October 20, 2025 @03:33AM (#45640887)
Thankfully Slack is still holding up.
Re:Tell HN: AWS us-east-1 services are down by LostMyLogin on Monday October 20, 2025 @04:13AM (#45641159)
It’s acting up for me but wondering if it’s unrelated. Imagines failing to post and threads acting strange.
Re:Tell HN: AWS us-east-1 services are down by tomwojcik on Monday October 20, 2025 @04:41AM (#45641369)
Same. My Slack mobile app managed to sync the new messages, but it took it about 30 seconds, while usually it's sub 2 seconds.
Re:Tell HN: AWS us-east-1 services are down by rhdunn on Monday October 20, 2025 @04:45AM (#45641411)
Slack is having issues with huddles, canvas, and messaging per https://slack-status.com/. Earlier it was just huddles and canvas.
Re:Tell HN: AWS us-east-1 services are down by tubs on Monday October 20, 2025 @04:01AM (#45641058)
It’s super broken for me. Random threads no longer appear.
[dead]
Re:Tell HN: AWS us-east-1 services are down by philipallstar on Monday October 20, 2025 @03:55AM (#45641014)
And the post office still works, so ah, at least kidnappers can send ransom demands.
Re:Tell HN: AWS us-east-1 services are down by esskay on Monday October 20, 2025 @04:29AM (#45641287)
Looks like its gone down now