News for hackers. Stuff that matters.
Re:Slack – Degraded service affecting multiple features by darkwater on Friday June 28, 2019 @08:51AM (#20303724)
Or not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.
Re:Slack – Degraded service affecting multiple features by kkirsche on Friday June 28, 2019 @08:54AM (#20303747)
For anyone else curious, SPOF means single point of failure in this context.
Re:Slack – Degraded service affecting multiple features by LinuxBender on Friday June 28, 2019 @10:54AM (#20304887)
At my workplace we were told SPOF was too negative, so one of the engineers changed it to SPOS. Single Point Of Success.
Re:Slack – Degraded service affecting multiple features by winslett on Friday June 28, 2019 @08:57AM (#20303767)
I was thinking "single point of focus."
Re:Slack – Degraded service affecting multiple features by triplee on Friday June 28, 2019 @09:04AM (#20303841)
Why not both?
Re:Slack – Degraded service affecting multiple features by zymhan on Friday June 28, 2019 @09:35AM (#20304119)
That's not really how acronyms work, that doesn't convey the same meaning.
Re:Slack – Degraded service affecting multiple features by SquishyPanda23 on Friday June 28, 2019 @09:37AM (#20304142)
It makes sense to have a fallback method set ahead of time, as others have mentioned.

You won't have access to the things slack integrates with but at least you can talk.

I know some people use IRC as a low-dependency fallback channel.

Re:Slack – Degraded service affecting multiple features by Twirrim on Friday June 28, 2019 @09:55AM (#20304289)
At one stage when there was an outage about a year ago, folks at work were scrambling around trying to get HipChat resurrected (which we'd been using prior to moving to Slack).

I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.

Re:Slack – Degraded service affecting multiple features by cloverich on Friday June 28, 2019 @10:21AM (#20304546)
Do you have a recommended server / client? The last time I tried to use IRC (joining an existing chat) I actually had quite a bit of trouble, and ended up just giving up.
Re:Slack – Degraded service affecting multiple features by notnull on Friday June 28, 2019 @10:39AM (#20304730)
Inspircd is pretty easy to set up (as a server). Hexchat is a great client, Weechat is great for a commandline client. If you have server access you can also set up Kiwi which is a webclient, making it easy for those with less tech prowess to connect.
Re:Slack – Degraded service affecting multiple features by LinuxBender on Friday June 28, 2019 @10:57AM (#20304912)
For ages, I ran unrealircd [1] and anope [2] services. One could put TheLounge [3] (web front end with redis) in front of it for persistence. Not as easy as inspire to set up, but rock solid and feature rich.

[1] - https://www.unrealircd.org/

[2] - https://www.anope.org/

[3] - https://thelounge.chat/

Re:Slack – Degraded service affecting multiple features by geggam on Friday June 28, 2019 @10:39AM (#20304724)
irssi for a client

Freenode runs this one for a server.

https://github.com/freenode/ircd-seven

Re:Slack – Degraded service affecting multiple features by jdenoy on Friday June 28, 2019 @09:41AM (#20304177)
we use a mumble server as backup and parallel mean to discuss in the operations department... Sometimes talking all together using voice as if we were in the same office is much nicer than writing!
Re:Slack – Degraded service affecting multiple features by LinuxBender on Friday June 28, 2019 @10:59AM (#20304933)
Mumble / murmur [1] is great. I set one up for a family member in minutes. Super easy to deploy. People should use their RC releases. They are solid, sound quality is great and it scales really well. A small to medium VM can handle thousands of people.

[1] - https://wiki.mumble.info/wiki/Main_Page

Re:Slack – Degraded service affecting multiple features by pennaMan on Friday June 28, 2019 @08:57AM (#20303770)
Having a single communication medium is a SPOF no matter what name it has
Re:Slack – Degraded service affecting multiple features by ratsimihah on Friday June 28, 2019 @09:11AM (#20303889)
Isn't internet a single communication medium anyway?
Re:Slack – Degraded service affecting multiple features by bluejekyll on Friday June 28, 2019 @09:14AM (#20303923)
The internet is specifically designed to route around downed links such that there is no SPOF, that’s the theory anyway.

However, when major backbones go down it takes it a while to recover.

Re:Slack – Degraded service affecting multiple features by kiallmacinnes on Friday June 28, 2019 @09:25AM (#20304023)
Well, kinda! Just look at this recent CloudFlare blog post[1]. It's true, theres no Single Point of Failure - there are MANY points of failure ;)

[1]: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...

Re:Slack – Degraded service affecting multiple features by isostatic on Friday June 28, 2019 @10:29AM (#20304629)
The internet, like the road network is designed to cope with failures - even if a major problem leads to congestion.

We build services on top of that that don't cope with failures.

Re:Slack – Degraded service affecting multiple features by kiallmacinnes on Tuesday July 09, 2019 @12:17PM (#20393378)
Does this change anything though? The internet has many of these SPOF services. And even more AS's who through malice or fluke can take trigger a failure of those SPOFs + more.

At the end of the day, that sounds like a massive pile of SPOFs that just barely keep up..

Re:Slack – Degraded service affecting multiple features by jimktrains2 on Friday June 28, 2019 @09:15AM (#20303934)
Not really? The internet is literally an internet, i.e. a network that connects networks. You can nearly always find another connection that still works, be it a coffee shop or cellular.

Sure, having a single connection is a SPOF, but anyone living in an urban area and anyone with a cellphone (and not using the same cellular carrier for home internet) has more than one connection.

Re:Slack – Degraded service affecting multiple features by beatgammit on Friday June 28, 2019 @01:21PM (#20306647)
I'm on Google Fi, so theoretically I have redundancy on my phone as well. I have used my phone as a hotspot for work, and I've also gone to other locations to get work done (library, coffee shop, a neighboring company's lobby, etc).

A lot has to go wrong for me to not get work done. If Slack goes down, I use text, email, or Jira comments (in roughly that order of priority). Relying on only one communications method is foolhardy.

Re:Slack – Degraded service affecting multiple features by dillonmckay on Friday June 28, 2019 @09:14AM (#20303929)
What OSI layer are we discussing?
Re:Slack – Degraded service affecting multiple features by samstave on Friday June 28, 2019 @10:30AM (#20304634)
Interestingly that you say this; we need an additional layer called layer 8.

As app is 7 but we have grown beyond the seven into meta layers of service types which have become the favric of how we operate... such as email, txt, chat etc to keep all other aspects of infra up and running.

Re:Slack – Degraded service affecting multiple features by kazen44 on Friday June 28, 2019 @10:48AM (#20304818)
the OSI model is not a truthful representation of the different layer of networking. Things get messy between layer 2 and 3 (ARP for instance?).

Also, everything above layer 4 is a enormous mess, considering that stuff is usually happening at the application itself anyways. There are network protocols which have proper OSI seperation, but they are rarely used. (IS-IS in combination with CLNS for instance).

Re:Slack – Degraded service affecting multiple features by samstave on Friday June 28, 2019 @11:06AM (#20305026)
Knowledge atrophy is a serious thing... i used to be deeply skilled in all of that but attention is where the heart is... so ive literally forgotten so much abt networking over the years.

A failure of mine was to not keep a project portfolio runbook which i could look back upon.

Imagine a system where all your status reports were logged and searchable - that would be move valuable than linkedin...

Re:Slack – Degraded service affecting multiple features by dpcx on Friday June 28, 2019 @09:12AM (#20303910)
It is, technically. Good thing there's also phones and SMS.
Re:Slack – Degraded service affecting multiple features by TeMPOraL on Friday June 28, 2019 @09:59AM (#20304320)
Both of which increasingly work over the Internet on the backend, from what I hear.
Re:Slack – Degraded service affecting multiple features by wbl on Friday June 28, 2019 @09:24AM (#20304014)
I have heard a horror story involving a phone line multiplexed over fiber to an IX and someone used it as OOB.
Re:Slack – Degraded service affecting multiple features by pfortuny on Friday June 28, 2019 @09:14AM (#20303921)
But it is routed. Also your laptop usually has 2 Network cards on different physical layers... Not the same.
Re:Slack – Degraded service affecting multiple features by duxup on Friday June 28, 2019 @10:17AM (#20304485)
It depends.

I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down.

I needed to do some things so I just stopped checking with people. So did other people.

The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it.

After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed.

Granted, this requires people to make good choices.

Re:Slack – Degraded service affecting multiple features by icedchai on Friday June 28, 2019 @09:26AM (#20304036)
Good thing email still exists, right?
Re:Slack – Degraded service affecting multiple features by majewsky on Friday June 28, 2019 @08:56AM (#20303760)
Same here. I'm on support duty and have my own outage to attend to, and that's really really hard when you cannot communicate with your experts.
Re:Slack – Degraded service affecting multiple features by wastedhours on Friday June 28, 2019 @09:17AM (#20303956)
I don't want to sound snarky by questioning what's probably an org mandated tool usage, but I can't quite believe companies where there's only one accepted way to get in contact.

Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).

Re:Slack – Degraded service affecting multiple features by jt2190 on Friday June 28, 2019 @09:00AM (#20303794)
Forgive me, but just to clarify... There is no “fallback” communication channel for your team to use?
Re:Slack – Degraded service affecting multiple features by wil421 on Friday June 28, 2019 @09:11AM (#20303891)
There’s email, phone, whats app, group SMS, some form of WebEx or Zoom, or even a shared google doc spreadsheet with tasks for each outage.

If Slack is your only forum of communication with your team it’s time to rethink your support structure and DR plans.

Edit: It’s very unhelpful if I point out issues and don’t provide solutions. PagerDuty has a great article about Incident Response[1].

[1]https://response.pagerduty.com/

Re:Slack – Degraded service affecting multiple features by matwood on Friday June 28, 2019 @10:32AM (#20304653)
I know it's 'old school', but we would all just get on a conference call before the fancy chat tools.
Re:Slack – Degraded service affecting multiple features by orwin on Friday June 28, 2019 @10:28AM (#20304605)
Yeah, i understand. hosting your own mattermost (or other slack clone) is not enough for you? You can also have 2 self-hosted solution on different servers (one on our own, one on any CSP), in case your server fails.
Re:Slack – Degraded service affecting multiple features by swebs on Friday June 28, 2019 @09:13AM (#20303917)
Why can't you use email?
Re:Slack – Degraded service affecting multiple features by StreamBright on Friday June 28, 2019 @09:12AM (#20303908)
That is rather unfortunate if there is only a single channel that an organization can use in a production outage situation.
Re:Slack – Degraded service affecting multiple features by darkwater on Friday June 28, 2019 @09:36AM (#20304126)
Well, there are other means of communication but they either:

* do not fit well for a quick message interchange like you can have during an outage (email)

* don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it)

* make compliated to share big chunks of text/logs (phone call)

It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly).

Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.

Re:Slack – Degraded service affecting multiple features by matchbok on Friday June 28, 2019 @09:30AM (#20304065)
Do phones exist?
Re:Slack – Degraded service affecting multiple features by pmlnr on Friday June 28, 2019 @09:00AM (#20303795)
This is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.
Re:Slack – Degraded service affecting multiple features by ibero on Friday June 28, 2019 @09:03AM (#20303831)
The commenter above said they had to "solve live production problems", how exactly would "pre-defined tasks" help them?
Re:Slack – Degraded service affecting multiple features by pmlnr on Friday June 28, 2019 @09:07AM (#20303861)
I slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.
Re:Slack – Degraded service affecting multiple features by wastedhours on Friday June 28, 2019 @09:13AM (#20303913)
I'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues.

If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).

Re:Slack – Degraded service affecting multiple features by dasil003 on Friday June 28, 2019 @09:38AM (#20304145)
What an incredibly arrogant generalization to make. You know there are also production issues created by moving too slow and failing to address pending scalability issues, right?
Re:Slack – Degraded service affecting multiple features by pmlnr on Friday June 28, 2019 @09:59AM (#20304323)
heh.

> failing to address pending scalability issues

How is that not related to not getting enough funding and resources?

And yes, having scalability issues means moving too fast - the _business_ moving too fast.

Re:Slack – Degraded service affecting multiple features by kaikai on Friday June 28, 2019 @11:07AM (#20305040)
This is starting to sound like the business version of starting physics problems with “imagine a frictionless surface...”