News for hackers. Stuff that matters.
Re:Slack was experiencing an outage by jrockway on Thursday May 20, 2021 @02:16PM (#27224968)
What practices do you use to ensure higher availability than Slack? Hope is not a strategy.

(Answering my own question, I'd guess that the strategy is "never change anything". To some extent, I don't need 80% of the features that Slack offers, and code that you don't have is code that can't break. Not restarting stuff to release new code is also good for uptime. Would be interesting to see a SaaS that has a super-stable-never-change-anything release track, hosted on infra with similar goals, and see how that uptime compares to people's homegrown version of the same strategy.)

Re:Slack was experiencing an outage by iso1631 on Thursday May 20, 2021 @02:54PM (#27225490)
My public facing servers (well https with client certificates that access them) are constantly upgraded, there's two of them, occasionally one goes down for a reboot overnight to get a new kernel, but clients will point to the other one.

Radius does vrrp, not instant, but acceptable

For true high availability (when downtime of milliseconds being unacceptable), we have A and B systems (sometimes more). E.g. syslog - router sends the messages out of two interfaces to two different servers which don't go down at the same time.

For local staff wanting access to internal webpages, we can cope with 20 second downtimes at an appropiate time.

I don't run a local irc server, but I'd be very surprised if it was any less stable than any other standard linux box -- do an apt or yum upgrade occasionally, any service outages you plan during a team meeting (thus we don't care if the chat system is down as we're talking via another method).

Stability is far easier when

1) Your server isn't open to the whole world

2) Your server isn't scaling to millions of users

Hell I've got a tcpdump session on one box that's been running since before the last slack outage 3 months ago.