News for hackers. Stuff that matters.
Re:Slack was experiencing an outage by pavel_lishin on Thursday May 20, 2021 @01:40PM (#27224389)
> "Hey guys, remember when _all of IRC_ went down?"

I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what.

I think the lesson here isn't "run and maintain your own version of X", but "make sure you have other ways to communicate besides X".

Re:Slack was experiencing an outage by outworlder on Thursday May 20, 2021 @01:50PM (#27224544)
> I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what.

But it certainly wasn't all the entire IRC network(s). Which is the parent's point. Slack is centralized.

It was a common practice back then to have one "server". You would be hard-pressed to find mentions of "high availability" as servers were a very scarce resource. Small networks sometimes repurposed existing machines that were serving other workloads and added the IRC daemon.

Re:Slack was experiencing an outage by andrepew on Thursday May 20, 2021 @02:13PM (#27224906)
From my perspective as a business, I don't see a difference between my server being down and all of IRC being down. Either way my comms are down.

All I care about as a business is about frequency and duration of those outages. If Slack outdoes whatever my in house solution is, Slack wins in reliability.

You can make arguments about the net impact on society of everything being down, but from my perspective as an individual business, having everything down is a win was well.

If I explain to client that we have a delay because our in house service is down -- fault falls on our business. If I explain to a client it is because Slack is down, odds are they'll blame Slack, not us. This is especially true if they're impacted by the Slack outage as well or the outage makes news outlets.

Outages from services likes Slack, AWS, etc. that are wide spread are almost treated like acts of god. Clients seem to be far more forgiving for those sorts of issues in my experience.

Re:Slack was experiencing an outage by _jal on Thursday May 20, 2021 @02:21PM (#27225035)
> If I explain to client that we have a delay because our in house service is down -- fault falls on our business.

This is an under-appreciated competitive advantage "cloud" services offer - outsourcing blame. Somewhat similar to using staffing agencies as lawsuit shields.

Re:Slack was experiencing an outage by throwawayboise on Thursday May 20, 2021 @02:33PM (#27225205)
No I don't think this argument holds water. Whose decision was it to use the centralized cloud service? Ultimately it's on you if your business can't respond to the customer.
Re:Slack was experiencing an outage by andrepew on Thursday May 20, 2021 @02:38PM (#27225266)
If the decision appears reasonable in the context of your industry, you won't be blamed.

Phrases like "Nobody ever got fired for buying Cisco/IBM/etc." are tossed around fairly often. If you chose the expensive, gold standard for your use-case and got burned anyway, odds are it won't be held against you.

Of course there are exceptions to that. If AWS going down results in loss of life -- you should take measures to avoid it. If the stakes are a day of lost work, the cure might be worse than the disease from a cost perspective.

Re:Slack was experiencing an outage by Spivak on Thursday May 20, 2021 @02:40PM (#27225303)
It's not about whether it makes perfect sense, it's about how understating your users are. People treat large cloud service outages like storms. When AWS has issues people aren't mad at me any more than they would be if they couldn't reach my brick-and-mortar store because of a flood.
Re:Slack was experiencing an outage by boardwaalk on Thursday May 20, 2021 @02:36PM (#27225235)
Will Slack actually outdo your in house solution though?

If you have a service (Slack alternative or otherwise) locally hosted, you're cutting out a whole bunch of failure modes:

  1. Upgrades/maintenance at inopportune times (you can control this, or not even upgrade at all if you don't need it)
  2. Networking issues, if you can colocate with with upstream or downstream dependencies
  3. Many scaling issues (disk fills up, etc) since you're only serving your own needs and you're not trying to put as many users in as little capacity as possible
  4. Relatedly, abusive use by other users won't effect you.
  5. You can keep the service off the public internet and avoid things like DDOS attacks
It's nice to do able to blame someone else, I get, but that seems kind of gross, and actually being able to provide the best service overall is what I'd go for.
Re:Slack was experiencing an outage by Hamuko on Thursday May 20, 2021 @02:39PM (#27225294)
And how many failure modes are you introducing?

If your service is off the public Internet, you now have a failure mode where the company VPN going down kills all productivity. And if you're going to have issues with a disk getting filled up, that's most likely going to happen with an on-premise solution than with a hosted solution where they have a team to take care that the disks don't "fill up". And not keeping your software updated because "an update would happen at an inappropriate time and we don't need it" sounds like a good vulnerability vector.

Re:Slack was experiencing an outage by iso1631 on Thursday May 20, 2021 @03:02PM (#27225593)
4 or 5 slack outages in the last 12 months.

Happy to do the update at a quiet time (say 3pm on a wednesday before we go to the pub), but wouldn't do it during a major event (say the middle of the world cup final when we're monitoring things very closely)

Re:Slack was experiencing an outage by jcims on Thursday May 20, 2021 @02:30PM (#27225173)
> If I explain to a client it is because Slack is down, odds are they'll blame Slack, not us. This is especially true if they're impacted by the Slack outage as well or the outage makes news outlets.

My perspective is skewed b/c I work at a place that invests 8-9 figures in their supplier risk management program, but in that domain you don't get any breaks here. This dependency is discovered during pre-contract due diligence and your plan and test history to continue operations in the event of a vendor outage is assessed in the overall risk profile of your company.

It's obvious this type of consideration in risk management is picking up steam and I'm expecting an industry to start forming around it. I'm also waiting for 'chaos engineering' features to start creeping into SaaS products.

Re:Slack was experiencing an outage by stickfigure on Thursday May 20, 2021 @02:03PM (#27224757)
Why does it matter to me if all of slack goes down at once?

I only interact with "my" slack. There is no material difference to my team's experience if "all of slack" goes down vs "our standalone chat server" goes down.

The only relevant issue is whether or not Slack's team creates a more reliable service than my poorly maintained homegrown solution.

Re:Slack was experiencing an outage by dijit on Thursday May 20, 2021 @01:45PM (#27224464)
I think that's a fair conclusion honestly, though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel.

Netsplits are mostly a solved problem these days, both with increased reliability of the internet and with network failure proxies like RobustIRC (for the linking protocol); and I remember fondly the times where we had a single IRC instance for all of our company (which had people working across: Thailand, USA and N.Europe) without issue.

But, yeah, I think what you said is completely fair.

Re:Slack was experiencing an outage by pavel_lishin on Thursday May 20, 2021 @02:10PM (#27224858)
> though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel.

Tell me about it! Two minutes after I posted that comment, I realized that our deploy system relies heavily on Slack.

Re:Slack was experiencing an outage by Hamuko on Thursday May 20, 2021 @02:11PM (#27224867)
>I realized that our deploy system relies heavily on Slack.

How? The only deployment integration we have for Slack is that we get a message when a deployment start and when a deployment ends.

Re:Slack was experiencing an outage by Spivak on Thursday May 20, 2021 @02:41PM (#27225321)
@deploybot do_release v2.3.45

I mean I've seen weirder. For Slack specifically I had a fight with a dev team because their deploy tool failed if it couldn't push a message to Slack.

Re:Slack was experiencing an outage by u801e on Thursday May 20, 2021 @01:46PM (#27224472)
I can't think of a time where the internal IRC server where I work went down. Netsplits shouldn't be an issue for that particular use case.