News for hackers. Stuff that matters.
Re:Slack is down by unethical_ban on Monday January 04, 2021 @12:03PM (#25634165)
I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it.

It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use Slack for system alerting.

Re:Slack is down by dysfunction on Monday January 04, 2021 @12:08PM (#25634278)
My coworker's theory was someone was waiting for the holiday's end to deploy something risky.

And I'm in that boat of depending on Slack for alerting... in fact my team was also waiting over the holidays to deploy more robust non-Slack-based alerting (in our defense the product is only a few months old and only now starting to scale to any real volume).

Re:Slack is down by bitbuilder on Monday January 04, 2021 @12:34PM (#25634842)
I wouldn't be surprised if it's actually a combination of a new feature being recently rolled out, along with the sudden spike in load this morning.

The holidays are actually the perfect time for Slack to roll out a risky deployment, as it has to be their lowest usage time. So it would make sense if something was pushed out last week or the week before. And everything probably seemed fine.

And then this morning they suddenly realize this new feature does not perform under load. And to make matters worse, the new feature has been out long enough to make any sort of rollback very tricky, if not impossible. Which means they'd need engineers to desperately hack out, test and deploy a code fix.

If this is the scenario, I do not envy them at all.

Re:Slack is down by rrrrrrrrrrrryan on Monday January 04, 2021 @12:59PM (#25635249)
Holidays are a good time for a company to do a risky deployment, but a bad time for an individual employee to do a risky deployment, assuming one doesn't want to work overtime over the holiday fixing things.
Re:Slack is down by spurdoman77 on Monday January 04, 2021 @02:02PM (#25636234)
Depends on how well compensated holiday overtime is. There are some employees happy to work overtime if their hourly pay is doubled or tripled. However there also those who wouldnt do that for any price.
Re:Slack is down by iso1631 on Monday January 04, 2021 @02:11PM (#25636365)
Depends how bad it goes wrong. My org is a 24/7 one, but one Christmas back in the 90s (way before my time) some work was done on Christmas eve, I think it was on the phone system, in the days before widespread mobile phones.

It broke, which was a major problem, this meant that senior management were being phoned (ho), and relatively high middle managers were on site to deal with the fall out. Of course most suppliers were also closed so everything was harder to fix.

There's good reasons not to do changes when places are closed, or at least skeletoned, for 2 weeks.

Re:Slack is down by ellisv on Monday January 04, 2021 @01:59PM (#25636184)
This depends on how easy/difficult the rollback strategy.
Re:Slack is down by hinkley on Monday January 04, 2021 @12:11PM (#25634362)
Not a bad theory.

I used to work for a place that had a FY that ended in summer. We had a lot less problems with stuff being shoveled out the door at Thanksgiving and Christmas because nobody was trying to finish their year-end performance goals over the Holidays.

I think what I'm implying is that management creates this issue, but we are complicit.

Re:Slack is down by spelunker on Monday January 04, 2021 @12:14PM (#25634426)
Yeah, I think it's this rather than load. Slack should be able to handle load fine (probably), but since this is the first weekday post-holidays I imagine some deployment broke something.
Re:Slack is down by t-writescode on Monday January 04, 2021 @12:54PM (#25635170)
Slack has been in business for several years and has survived several December to January transitions, including several people stopping using their product before Christmas and then returning early January.

It seems a bit presumptuous to assume that's at fault here, given their age.

Re:Slack is down by rrrrrrrrrrrryan on Monday January 04, 2021 @01:00PM (#25635264)
Does it? Don't you think their users might be leaning on it more heavily this year due to working from home?
Re:Slack is down by hinkley on Monday January 04, 2021 @12:08PM (#25634281)
The two cliched sources of this problem are 1) someone pushed something out over the holidays that could have waited until January, or 2) peak capacity was negatively affected since the last time a spike happened, nobody had a way to monitor it, and so this has been broken since the end of May. On further reflection, someone will admit that they noticed a notch-up in response times and did not connect the dots.
Re:Slack is down by djxfade on Monday January 04, 2021 @12:06PM (#25634246)
We get our primary alerts through Slack. However we also have SMS and phone call backups through PagerDuty
Re:Slack is down by CSDude on Monday January 04, 2021 @02:10PM (#25636356)
(from Opsgenie) I would imagine it would be the other way around for most people.
Re:Slack is down by djsumdog on Monday January 04, 2021 @12:10PM (#25634328)
We have an alert channel in Slack, but it's mostly ignored. Our primary alerts come via SMS/VictorOps.

At one of my old jobs, we had SMS via two physical/hardware devices in our data center. One had a Telstra SIM card and the other had an Optus SIM card. (They were plugged into the same machine, but we had plans to put a second one in another data center before I left).

If you really care about alerts, you should have physical hardware doing your SMS messages via two different point-of-presences.

Re:Slack is down by twblalock on Monday January 04, 2021 @03:01PM (#25637050)
Now is a good time to recommend to your engineering org that they should have multiple alerting methods, e.g. Slack plus Pagerduty, or Slack plus email.

Hopefully email won't be your backup. I've seen that done. Alerts get filtered and ignored, often by accident.

Re:Slack is down by tqi on Monday January 04, 2021 @12:52PM (#25635140)
Do you know that was the root cause or are you making an assumption and running with it?
Re:Slack is down by raverbashing on Monday January 04, 2021 @12:08PM (#25634296)
It might have been the increased usage due to the pandemic (since it didn't happen from 2019 to 2020) + the sudden inflow of people at the same time.
Re:Slack is down by majewsky on Monday January 04, 2021 @12:50PM (#25635116)
This would be a good explanation for an outage in March or April 2020, not so much in January 2021.
Re:Slack is down by T-hawk on Monday January 04, 2021 @01:02PM (#25635302)
There may well have been a bigger delta in usage from Sunday Jan 3 to Monday Jan 4 2021, than for any particular pair of days in March-April 2020.

Of course 2020 saw an increase, but it was smeared over a week or a month rather than being a big jump in a single day after everyone's holidays.