To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases:

- high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, the higher the divergence world of production and the world of the new feature

- lots of new hires (new year = new hiring budget). New hires are missing some tribal knowledge about the system and make a production-breaking release.

I tried to think of other reasons, but these two overwhelmingly stand out as the two biggest reasons. Would love to hear from others.

If new hires tends to break production, it's not in the first business day of the calendar year. December gets really quiet for recruiting, typically, as candidates get busy with their social lives, and scheduling interviews gets harder.

January is busy for recruiting, but given a week or two of interviewing and negotiating, two weeks notice, it's probably February before new employees are starting, and they're not making big, production-damaging deploys for a week or two after that.

You will also get a pause in new hires in late December for the same reason. I've certainly accepted an offer late in the year and then didn't start until the new year.

Probably not as big of a rush as the end of school year rush in summer though.

I also doubt that new people will be breaking production on day one. Even at a fast moving startup I'd expect it to take a bit to go through the onboarding paperwork, get a laptop and actually try pushing a change to production.

I think some big company (maybe Facebook) has this rule that you had to deploy something to production on your first day. They seemed pretty confident in their processes and devops teams. A company trying to imitate that policy without doing the work necessary to make it possible would probably have outages on days when lots of new people joined :-P

Could be Facebook as I think production releases are always rolled out in phases e.g. first to 10 users, then 100, then 1000 and so on. That means there's much less chance of even the worst mistake having a serious effect.

Wow, onboarding new hires here is going good, if they can access slack, O365, LDAP, VPN and clone the repo by the end of the first day. Tho we have the initiation ritual of installing the OS to your laptop.

Sudden surge of traffic as all their users returns to work?

Could be, it's the perfect time overlap between US-West, US-East, and Europe.

Yes - I wondered if they took some servers down prior to the break as a cost saving measure, and forgot to reinstate them.

Doubtful. It's not impossible a company the size of Slack would be reliant on a specific engineer logging on in the morning before a traffic spike so the service can handle the spike in load, but that's a misuse of modern distributed cloud-based computing.

Hate on the cloud all you want, but AWS has (several flavors of) load balancers and various ways to automatically scale up and down resources (and if you're conservative, you can disable the 'down' part). If you're operating a major SaaS company like Slack and not taking advantage of them, something's gone wrong.

It's easy to fall behind on bumping up the high watermark for your max autoscaling or for new traffic patterns to cause emergent instability. New code paths are taking unprecedented amounts of traffic all the time.

In 2021, how does one keep track of resource starvation at the process, container, os, service, pod, cluster, availability zone and region levels?

I would add here the potential scaling issue - holidays were a dry season - less meeting. So if they have some automation for scaling down to reduce cost, it may have bitten them in their arses now.

People came back to work, and most of them start around the same time (US wise at least).

Hence kids - a vital lesson for all of us - don't start the call at a full hour, give it 3-7 min to make your coworkers confused and give some time for the systems to auto-scale ;)

I hope this isn't the case. It's not like this is the first holiday season for slack.

I think you're right on the first bullet, but not the second. If it was mid-Feb, then maybe, but the next FY hasn't even started yet for a ton of companies, let alone onboarding newbies to production.

People returning to work and downloading a huge backlog of messages from the past two weeks.

As an Microsoft Teams Ex-Dev, I can vouch that message retrieval after vacation puts a lot of stress on storage systems before it stabilizes :)

Yeah, makes sense. A system typically optimized for performance and real time delivery is suddenly asked to perform multiple batch retrievals in large chunks. Ouch!

I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods.

Doubt anyone releasing big changes Monday morning.

I haven't worked at Slack, so I can't speak with high confidence. A traffic spike is a possible reason, but I'm willing to bet that it's not the reason:

> Doubt anyone releasing big changes Monday morning.

This is definitely an engineering best practice, and by best practice, I mean something that Uber's, I mean Slack's SRE team strongly pushed for, and got politely overruled on. After a code freeze is lifted, it's quite common for lots of promotion-eager engineers to release big changes.

In my experience it's not promotion-eager engineers that want to push after a code freeze, it's antsy product managers. YMMV tho.

IMO it really doesn't have to be promotion-eager engineers or antsy product managers. I'm fairly satisfied with my role and comp and work type with where my career/life-stage is. I just did a code release first thing this morning, not because I am promotion-eager, but just because I'm picking back up where I left off, like any normal day. Granted I work at a much smaller company than Slack with orders of magnitude less traffic.

What's there to change in Slack, though? It's arguably a messaging system, and that feature is tried and tested. That, and giphys, to be honest.

EDIT: Guys it was a joke, chill

HN's tolerance for jokes and sarcasm is extremely low.

I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.

Yeah, Poe’s law applies here. That’s definitely something someone less informed might say in earnest.

Yeah, there was other thread about Uber, where similar sentiment was seriously debated there, so I didn't recognise this as sarcasm either.

You just don't deploy something major the first day after a 2 weeks vacation, it does not makes any sense.

Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.

Well, I think it probably depends on where you work. At my work, people just took 2-3 weeks of time off. It takes a moment to get your head back in the game.

Yeah, I do this all the time. I don't want to be bothered on the weekend, so I push releases at the beginning of the week when possible.

Same, I would rather release on a Monday than a Friday.

Everywhere I've worked often has a massive backlog of things that get released after a moratorium or extended holiday week. Those are usually the worst weeks to be oncall since things are under so much churn.

8am in the first day is too early... But by 10am, after catching up on emails, it's totally time to start releasing stuff.

It depends on the goal you’re trying to accomplish. Are you going for a promotion or bonus? Or instead is your goal to maximize uptime?

I doubt that regularly releasing breaking changes that reduce uptime is a good strategy to get a bonus or promotion.

Assume promotion after releasing 10 changes

Releasing 1 change a year with a 100% chance of working -- no promotion for 10 years

Releasing 10 changes a year each with a 10% chance of breaking something -- 1 in 3 chance of promotion in a year, and a 2 in 3 chance of downtime

> Assume promotion after releasing 10 changes

Where did that assumption come from? Also are you claiming that it takes 10x more time to release a non-breaking change?

Interesting, I've never worked anywhere where engineers decide when to release changes. That's a product decision, and there is a process of review and approval at both the code level and the functional/end-user-experience level that has to happen first.

Did you mean that literally? E.g. is it common at Uber that engineers can release changes to production on their own?

At Cisco (Webex team), the engineers decide when to release code, and most features are enabled by configs or feature flags independently of the deploys.

The engineering team is responsible for the mess caused by a bad deploy, so it's appropriate that those engineers should also choose the timing.

Our team typically deploys between 10am and 4ish, local time, since that's when we're at our desks and ready to click through the approvals and monitor the changes as they go through our pipelines.

The feature enablement happens through an EFT / beta process, and the final timing of GA enablement is a PM decision. But features are widely used by customers ahead of that time, as part of the rollout process.

Our team usually rolls out non-feature changes to services via dynamic configuration switches, so that we can get new bits in place, and then enable new behavior without a redeploy. This also enables us to roll back the dynamic config quickly if something unexpected happens.

(We generally don't do this for net new functionality; there's lower risk in adding a new REST endpoint etc. than in changing an existing query's behavior or implementation.)

Does Uber/Slack not release in CI/CD? At least in backend?

I don't see any need to deploy a big change at once in the software world today. At worst feature gate the thing you want to do and run it in a beta environment, but still push the actual code down the pipeline.

> run it in a beta environment

Every Uber/ex-Uber engineer is nervously chuckling at this comment right now

As the saying goes, Everyone has a Test environment. Some people are lucky enough to have that distinct from Production.

For those that don't know what this comment is about: https://eng.uber.com/multitenancy-microservice-architecture/

I'm actually more confused after reading that. I assumed that you meant that tested in production on purpose, but it sounds, at a skim, like they do non-prod testing environments - in fact, it looks like they've gone to having multiple beta environments of every service?

My understanding is that they have a "tenancy" variable in every service call which can take a different code path. They seem to only have one environment for everything and do tests/experiments at code level based on this variable.

Ah, thanks; that explains it nicely

Aaah the wonders of not having to be PCI or SOC2 compliant...

That might be true but when you take the global usage of Slack and their respective time zones, more than half the world would have signed into Slack this morning before SV had and I certainly didn't notice any downtime this morning in my time zone.

It was ropey before SV woke up, I thought it was just my (normally rock solid thanks to using Ubiquity) network having issues.

Guess it was Slack being Slack.

What would make that strange? Where I work it is frowned upon to do releases on weekends and so bad changes due to buildups happen on Monday.

Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.

I have definitely worked in places where the times right before and right after a change freeze were the most unstable, so that could be it. However, as others have mentioned, it's pretty early on the west coast of the US. Unless some engineer was up extra early (perhaps at the behest of an anxious project manager) it seems unlikely to be a release.

What it could be is some engineer somewhere coming in after the holiday, noticing a slightly flaky thing, and thinking, "I'll reboot/redeploy/refresh this thing so the flakiness doesn't get worse". Only it turns out the flaky thing was a signal of something else falling over. Or maybe the redeploy was the wrong version because of bad CI/CD, or maybe the person just fat-fingered it.

Most releases are automated with time lockouts.

In what companies?

Competent ones like those you'd hear about being down on HN.

At least that how it worked at one FAANG

It varies a lot by team... I think it's common to have a single click "start" button to press. It's a good sanity check that a release isn't going to happen during a fire drill, outage, or strike...

Or unless that engineer was not in the US

Very possible. I don't know what Slack's workforce distribution is. In places I've worked there have definitely been some incidents in US off-hours triggered by someone on the other side of the world.

Another common cause is resource exhaustion as a result of poorly monitored resources (or bugged monitoring). For example Google's authentication was down because their system reported (wrongly) available quota of 0. The last two incidents at my company were also related to resource exhaustion.

This is one of the original concepts why to go capital-A Agile. Make smaller releases more often, so at least if something breaks, it's (hopefully) something small, and least it's easier to trace.

(I'm not making a statement if that's good or bad or if it works or whatever. Please don't read an opinion into it.)

This. If you roll many changes into a single deployment, you don’t know which change broke what. But if you have two or three weeks of commits waiting, it’s hard to do otherwise.

That's why good regression tests and CI are so important; in an ideal world (which we were close to in one of my projects), every change is pending in a pull request; the CI rebases the change on top of its upstream (e.g. master/main), simulating the state the codebase will be in once merged, and runs the full suite of tests. The build is invalidated and has to be re-run if either the branch or upstream is changed.

Now, caveats etc, this was a collection of single applications in a big microservices architecture, and as the project grows it becomes more and more difficult to manage something like this, especially if you get more pull requests in the time it takes to do a build. But it is the way to go, I think.

Anyway, since tests and CI are not definitive, you also need a gradual rollout - 1%, 5%, etc - AND you need a similar process for any infrastructure change, which gets more and more tricky as you go down to the hardware level.

This is very likely a broken release. The timing lines up with pacific time too well.

They declared the issue at 7:14AM PST. How long is their deploy process?

That sounds pretty early to think somebody on the west coast did something, other than maybe acknowledge the pages and declare the incident.

Slack does progressive roll-outs. The broken release hypothesis seems very unlikely.

If you think about it, modifications to state of the system caused by human actions are the sole purpose of computers.

Seems to be more than that. Even slack.com in an incognito browser fails.

What does an incognito browser have to do with anything?

An incognito browser would ignore all client-side cookies, so the Slack web client would not try to - say - resume a previous user's session or re-use any previously saved data.

Likewise, incognito mode will also ignore most cached web content, meaning all assets on the Slack web app will get loaded again from scratch. This "clean state" start could, theoretically, get around issues with old - potentially incorrect/outdated - assets being loaded, even though that really shouldn't happen under most circumstances.

Sure, but why does that indicate the issue probably wasn’t related to a code push, like the person I responded to said?

It means that one is not sending a session cookie of any kind, thus should be sent to a 100% cached version. No "Are you XYZ and what to log into ABC's Slack again?" box.

That means it's not a user auth error

non-logged in user may not go through all the same codepaths as a user with cookies present.

This is why Change Management is the main tenant of principles like ITIL