News for hackers. Stuff that matters.
Re:Slack is down by schoolornot on Monday January 04, 2021 @10:26AM (#25632475)
How many more outages until all trust is eroded and competing services differentiate themselves on the basis of uptime?
Re:Slack is down by jrockway on Monday January 04, 2021 @10:41AM (#25632714)
I say this every time Slack is down, but they just seem so shady to me. Nobody can connect right now, and their status site says "100% uptime in the last quarter". Maybe it's close to 100%, but it ain't 100%.

I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service. If 1% of users can't send messages, then that should count as a full-blown outage and should start counting against whatever SLA they advertise.

The underlying problem here is that apparently everyone lies about uptime, so if you don't, that looks bad to potential customers. I fear that we will have to push for some legal regulation if we want accurate data, and ... people will probably be opposed to that.

Re:Slack is down by gwright on Monday January 04, 2021 @12:11PM (#25634354)
Seems silly to worry about quarterly stats several hours into an outage. The most obvious explanation is quarterly stats aren't generated in real-time -- which isn't "shady" to me.
Re:Slack is down by derefr on Monday January 04, 2021 @10:58AM (#25632955)
> I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service.

I mean, that’s nice to say, but how do you measure/prove it?

Certainly, having the SLAed party check themselves is silly. But what are the other options? If it was up to the customer, customers could make up faults to get free service. (Since it’d be up to the customer to prove, and customers are generally less technical than vendors, you’d have to expect/accept very non-technical — and thus non-evidentiary! — forms of “proof”, e.g. “I dunno, we weren’t able to reach it today.” Things that could have just as well been their own ISP, or even operator error on their side.)

IMHO, contractual SLAs should be based on the checks of some agreed-upon neutral-third-party auditor (e.g. any of the many status/uptime monitoring services.) If the third party says the service is up, it’s up in SLA terms; if the third party says the service is down, it’s down in SLA terms.

(And, of course, if the third party themselves go down, or experience connectivity issues that cause them to see false correlated failures among many services, that should be explicitly written into the SLA as a condition where the customer isn’t going to get a remedial award against the SLA, even if the SLAed service does go down during that time. If the Internet backbone falls over, that’s the equivalent of what insurance providers call an “act of God.”)

But in a neutral-third-party observer setup, you aren’t going to get 100% coverage for customer-seen problems. An uptime service isn’t going to see the service the way every single customer does. Only the way one particular customer would. So it’s not going to notice these spurious some-customers-see-it-some-don’t faults.

So, again: what kind of input would feed this hypothetical “100% of customers are being served successfully” metric?

ETA: maybe you could get closer to this ideal by ensuring that the monitoring service 1. is effectively running a full integration test suite, not just hitting trivial APIs; and 2. if gradual-rollout experiments ala “hash the user’s ID to land them in an experiment hash-ring position, and assign feature flags to sections of the hash ring” are in use by the SLAed service, then the monitoring service should be given N different “probe users” that together cover the complete hash-ring of possible generated-feature-flag combinations. Or given special keys that get randomly assigned a different combination of feature-flags every time they’re used.

Re:Slack is down by teraflop on Monday January 04, 2021 @11:59AM (#25634086)
> If 1% of users can't send messages, then that should count as a full-blown outage and should start counting against whatever SLA they advertise.

Google published a paper last year describing this approach to measuring uptime: https://blog.acolyer.org/2020/02/26/meaningful-availability/

The idea is to define availability as "the probability that the site 'appeared' to be down for a random user, averaged over a time window of size w". You can choose a particular value of w and look at trends over time, or you can plot availability as a function of w to understand patterns of downtime.

Re:Slack is down by de_Selby on Monday January 04, 2021 @10:49AM (#25632827)
They should at least update the status site to reflect issues currently happening.

I was wondering why the link from a Jira wasn't opening in slack, the page eventually timed out and gave me a link to status.slack.com where it told me everything was peachy. Cue me wasting time trying it again because apparently there was no issue with slack..

Re:Slack is down by helper on Monday January 04, 2021 @02:39PM (#25636767)
That number is almost certainly updated manually. Check back tomorrow and see what it says.

If you look at the history page you can see its not 100% for every month: https://status.slack.com/calendar

Re:Slack is down by helper on Wednesday January 06, 2021 @12:59PM (#25660310)
(2 days later) The status site now shows %99.9 uptime for the quarter.
Re:Slack is down by swsieber on Monday January 04, 2021 @11:16AM (#25633274)
Some companies do this, though probably not publishing data. Any customer downtime is treated the same - for one, for many, for all (in theory, ha ha). But they take it pretty seriously.
Re:Slack is down by nkassis on Monday January 04, 2021 @10:52AM (#25632866)
You'll just end up with no SLA or pay a hefty amount to use services because that's an impossible standard to support for any service of a size like this.
Re:Slack is down by scrose on Monday January 04, 2021 @10:58AM (#25632948)
Isn't this the problem? Companies like Slack set SLA's that they only meet by lying about their uptime. It's as good as having no SLA, except you're likely paying a premium based on the SLA they set.
Re:Slack is down by jrockway on Monday January 04, 2021 @12:30PM (#25634756)
I'm not demanding 100% uptime, I'm asking that they say "99.94% uptime" when there has been an outage.

Honestly, I could live with a 99.50% SLA, if that's what it really was. After today's probably full-day outage, they'd just have to be extra careful for the rest of the year (or pay me money). Kind of sucks when it's 1/4 that you blow your year's SLA budget though.

Re:Slack is down by dijit on Monday January 04, 2021 @10:31AM (#25632543)
If you're asking genuinely then I can tell you my experience when I was part of a SaaS shop, though the times have changed a lot and "my metric is not necessarily your metric".

But it was roughly "one large impact a month, for six months", with large caveats that upper management for whatever company had to be working with the product during that month.

Large companies don't care if X service went out during the night and impacted someone not in their timezone.

If the CTO notices that he can't use something with the same regularity that he gets paid, then it doesn't take long for it to stick in their mind. But migrating everything is _so painful_ that the majority of large companies will do anything they can to avoid moving away.

Re:Slack is down by xibalba on Monday January 04, 2021 @10:35AM (#25632616)
> But migrating everything is _so painful_

This is a key point is the popularity amongst VCs in investing in B2B SaaS. I take their (and your) word for it. But honestly, I don't actually understand this.

Why is migration so hard?

Re:Slack is down by danpalmer on Monday January 04, 2021 @10:41AM (#25632720)
Medium sized team on Slack. We'd need to move ~60 full time in-house employees, ~10 remote contractors who aren't on other comms channels, ~20 infrequent freelance contributors who may not check messages often, ~5 custom bots and apps, and ~15 3rd party integrations (of which some won't support any given choice of alternative).

This is not to mention the fact that half our staff aren't hugely technical, so have actively _learnt_ how to use Slack and it's features around notification control (things that may come "naturally" to the tech-savvy crowd on HN), @-things, bots, etc, and they would need to re-learn a new tool that is going to work in a different way.

This would be a substantial effort for us, and we're a small company. Are there ways to materially minimise this cost?

Re:Slack is down by Frost1x on Monday January 04, 2021 @10:44AM (#25632759)
Training, integration with proprietary internal systems, sheer momentum in the employee base, justifying or even creating a metric to show cost savings of a migration effort, business processes that rely on a specific feature of existing infrastructure needing to be met, the uncertainty of new vs the certain and known instability of something you have....

If you had a small shop with a dozen tech-savvy people and Slack became a problem which was used exclusively for quick business chats, you could probably push a change to another chat platform the next day. You might struggle when you have thousands of employees, some that needed training to use Slack and still aren't that proficient.

Re:Slack is down by dhagz on Monday January 04, 2021 @10:43AM (#25632751)
Getting workflows re-established, any integrations you had developed or otherwise come to depend on may not work, you will probably lose history, etc.

Plus, it will just take a long time to get everyone on board and using the replacement system. My department is slowly plodding towards using Teams over Slack, but there are enough hold-outs (my sub-department being one of them) that it still doesn't have wide-spread adoption.

Re:Slack is down by twh270 on Monday January 04, 2021 @10:57AM (#25632931)
Many reasons, almost none of them technical. Off the top of my head, a few:

* Getting out of the Enterprise Contract, or waiting for the year to end. * Training people on new software. * Loss of productivity. (1) Learning a new UI, processes, workflows -- both individually and organizationally. A feature or concept in "Tool A" may exist in a completely different form in "Tool B". Or not exist, and then people need to adapt to and work around the missing feature. (2) Missing out on needed information due to the above. Ultimately, software exists to move and transform data, and when you change the software people have to adjust. Sometimes that doesn't go great. "Oh, I didn't realize I needed to check this checkbox".

Another way to say this is "organizational inertia", which is a fancy term that means "it's hard for people to adjust to change".

And you might think developers and other technical people would have an easier time of it. They (we) do, but not to the extent you may expect. I've been on the front lines of a handful of migrations that affected only the IT staff, and it was a long and arduous process each time.

Re:Slack is down by tpxl on Monday January 04, 2021 @11:27AM (#25633493)
> Loss of productivity. (1) Learning a new UI

Man it bothers me so much when applications change their UIs on updates for no apparent reason other than "it looks better".

IntelliJ changed the way build and debug buttons looked in some update and it took me days to get used to it and I could find them in a snap again. Slack did a couple of no-reason changes as well.

Re:Slack is down by gen220 on Monday January 04, 2021 @11:44AM (#25633801)
There are plenty of UX reasons (learning new interfaces, etc). The burden here is generally distributed and diffuse.

The really big one, for companies of a certain size / cash flow, is compliance. Companies spend a lot of time developing compliant work flows around a service like Slack.

Migrating to another service requires rewriting the compliance narrative. The current compliance people might not have the confidence or willpower to do that effectively, and can raise legal objections to any such migration indefinitely.

Re:Slack is down by f0ff on Monday January 04, 2021 @10:41AM (#25632724)
>Why is migration so hard?

Why would anyone make it easy

Re:Slack is down by dijit on Monday January 04, 2021 @10:51AM (#25632848)
I think there should be a new computer science law (if this one doesn't exist already):

Things that are easy to migrate from get replaced by things that are hard to migrate from, eventually.

IRC is incredibly easy to migrate from.

Re:Slack is down by ludjer on Monday January 04, 2021 @11:24AM (#25633445)
IRC is easy to migrate from since there is nothing to migrate other then chat history. IRC is also missing so many features that slack provides out the box. And a law like that would not work since you would need to write complicated transformation scripts to transform between services. Also not all services are a 1-1 mapping. I like IRC but it has its limitations. That is why slack succeeded where IRC did not.
Re:Slack is down by tpxl on Monday January 04, 2021 @11:28AM (#25633512)
> And a law like that would not work

The parent meant a law as in "a law of physics", not a piece of legislation.

Re:Slack is down by x86_64Ubuntu on Monday January 04, 2021 @11:40AM (#25633734)
We can call it the "Law Of Lotus Notes". I'm not sure if it's hard to migrate from, I can only assume that it is impossible to migrate from.
Re:Slack is down by ceejayoz on Monday January 04, 2021 @10:28AM (#25632506)
I wouldn't be shocked if businesses saw increased productivity during these.
Re:Slack is down by saxonww on Monday January 04, 2021 @10:42AM (#25632726)
I have less of an excuse not to be more personally productive, but I can't help anyone else (easily) if my primary method of communication is down. Not only because it's harder to contact you, but also because it's impossible for you to just ask in a channel and have me notice you.

There's also this perverse incentive to Slack all the things. Lots of CI notifications are sent through it. Some org processes are implemented as workflows. There's been talk of how wonderful it would be to hook up tasking and work tracking to slash commands. I and others often use Slack instead of the 'official' tool to video call each other.

An outage like this is still really disruptive. It's not like everyone realizes what's going on immediately or at the same time; we have backup tools, but our turn radius is pretty wide. Some of us can't even communicate effectively without memes, too, and backup tools don't have a giphy integration.

EDIT: Do your CI integrations fail if Slack can't be contacted? Do those failures fail your pipeline? Whoops!

Re:Slack is down by ghostpepper on Monday January 04, 2021 @11:12AM (#25633215)
Particularly on a Monday morning after a holiday, there are tasks that I know I need to be working on but cannot because relevant details were never transposed from slack to our actual work scheduling tools like google docs, jira, etc. and I cannot access Slack history.
Re:Slack is down by postalrat on Monday January 04, 2021 @10:37AM (#25632649)
I'm sitting here not sure if I should deploy code since most communication with the rest of my team has been cut off.
Re:Slack is down by derwiki on Monday January 04, 2021 @11:05AM (#25633088)
If something went awry, and it caused more pain because Slack was down, how would you feel? If you’re missing comms/observability then waiting to deploy seems prudent.
Re:Slack is down by dbbk on Monday January 04, 2021 @11:10AM (#25633171)
So the answer is... people should stop working?
Re:Slack is down by derwiki on Monday January 04, 2021 @11:31AM (#25633575)
I can't speak for you, but I can:

  * work on code
  * update JIRA
  * complete required trainings
  * work on my peer reviews (Workday/Okta are up)
  * review tech specs
Deploying is actually a very small part of my job.
Re:Slack is down by postalrat on Monday January 04, 2021 @11:14AM (#25633245)
Of course there are other things to do. But the things I had planned for the morning are all being delayed.
Re:Slack is down by falcolas on Monday January 04, 2021 @10:33AM (#25632572)
Not all - many workflows these days rely on Slack or its ilk. Benderbot, Jira/etc. connectors, calendar connectors, remote communication/standups, alerting…
Re:Slack is down by ryanSrich on Monday January 04, 2021 @10:52AM (#25632871)
If you use slack primarily as a water cooler then yes.

However, I drive everything through slack - GitHub, linear, calendars, Notion, support emails, etc. I have notifications turned off for every service we use except for slack. This allows me to effectively ignore everything except for slack. These types of outages destroy that workflow for me.

Re:Slack is down by dbcurtis on Monday January 04, 2021 @12:15PM (#25634448)
Absolutely! Before the holiday shutdown, I Slacked myself a huge reminder list of things to jump on as soon as we started up again, so that I could hit the ground running in the new year. Oh, wait....
Re:Slack is down by dkdk8283 on Monday January 04, 2021 @10:32AM (#25632567)
I feel a little good every time Slack has an issue - it has brought social media communications to my otherwise social media free (excluding hn) life.
Re:Slack is down by NewEntryHN on Monday January 04, 2021 @01:57PM (#25636167)
You seem to ask the question about the absolute number of outages, whereas uptime is about the number of outages per units of time.
Re:Slack is down by Thaxll on Monday January 04, 2021 @10:30AM (#25632533)
None because all competing services have some problems at some point.