There's a reason the military does drills.

Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on).

Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur.

Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.

>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems.

All those man hours cost money. Is it worth it? Depends on how much downtime costs you.

Most companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage per week.

It's also working out which outages matter the most. A 10 day outage is not equivalent to 10 1 day outages, the latter has a multitude of important reputational risks - but the former is quite possibly enough to end the business completely.

Depends on how you value the potential outcomes. A while ago, I think it was Gitlab that didn't check their backups and they were a dud.

That caused quite a few problems...How much is it worth to them after the fact? I'd say a fair bit.

But hindsight...

If you run a fairly discretionary web app, like perhaps a game where downtime will cost you lost revenue for the time period but that's it, it's probably not worth it.

If you are running a communications platform like Slack or Gmail, it's hella worth it.

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

But how, exactly?

Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production.

In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures.

It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.

> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question

No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.

Maybe, though again I would expect an alert for the process being dead would also not uncover this particular issue. It's possible someone would notice that there was insufficient alerting then go back to add something which would have caught this but it's far from certain. OTOH it's also possible that at this stage in the game, when the code and systems are EOL, that something like chaos testing is the last chance to catch this problem.

I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.

> Chaos Engineering

...or just "availability testing" or simply "testing".

Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices.

For example: saturating the uplink bandwidth on a whole datacenter.

To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.

I don't think that would have ever provoked this error mode, though.

It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.

It might have prevented this failure by killing stale HAProxy instances...

To catch this particular issue, they would have had to overprovision webapp instances beyond their N*M limit, take down a subset of the originally provisioned webapp instances and reprovision new ones in order to have observed this particular issue with their HAProxy setup.

It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.

This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.

Yep. I once worked at a company that had backups of all the data. Until they needed a backup and discovered they were not readable. Months of work were gone.

This has happened to me.

Long story. Incompetent (and dishonest) IT person. In fact, I was attacked for “harassing” the IT person.

Nothing was done about it, until the HR DB got borked, and there was no backup.

DR and backup are profoundly unpopular topics. They tend to be expensive, and difficult to test. They also presuppose a Very Bad Thing happening, which no one wants to think about.

I have scars.

I now have multiple layers of backup, and keep an eye on them. I can’t always test every aspect, and have to have faith in cloud providers, but I make sure that, even if we have a tornado, I would still be able to recover the important bits.

there's a joke in the data protection industry.

Backups always work, it's the restores that have problems

Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data.

We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right?

Big risk of having your privacy sensitive information stored on other infra than the main servers a path to get that information to your QA servers should not even exist.

Many more people that should not have access to that data, in bulk or otherwise suddenly have access which does an end run around all your - hopefully - carefully crafted security.

Keep in mind that what you think isn't privacy sensitive may very well be radioactive when combined with other data.

If you have a procedure for deriving QA data from production, change it to derive from the backup.

Using it directly is full of dangers, not only of leaking information, but also of corrupting your backups. (Otherwise, why are you testing anything?) And deriving test data from production looks like a good thing to me, but make sure to restrict the access to the test environment and mask your data.

You should not use production data for QA, period. Use generated data.

If you really must use real data make sure that (1) you have explicit direction to do this from those higher up in the food chain so that if there ever is a breach of your test systems you don't end up holding the bag and (2) anonymize a copy of the data before it leaves the production system. That way you minimize the risk. But better: simply don't do it, generate your test data.

Make sure you restrict access to the production environment, too. I once worked at a company where the "dev" office was VPN'd into production at all times. We actually didn't have a QA system for most of the time I was there. Anyway, a developer ran his server with the prod config for a couple of days before anyone noticed that requests were being processed elsewhere.

I think it is a big issue to use anything containing PII in anything else than production environment. For example, do your QA logs have the same kind of management, access control and lifetime than production ? If they don't, you might end up logging PII in a way you should not, and could even be in legal troubles for things like GDPR.

Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery.

In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more.

What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)

What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok?

For, let's say, a b2b saas

They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)

It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page.

It's up to the business to define SLAs and SLOs that the customer will be satisfied with.

> do it at the same time you'd schedule regular maintenance ... SLA ... SLO

I'll do that (some day), thanks

If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.

I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.

Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc.

In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions).

Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Yep, and Grafana allows alerting on No Data states.

One approach is to have two different monitoring checks that could catch the same root cause.

In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic.

It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger.

Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.

Chaos engineering.

You have to randomly simulate failure cases of all shapes and sizes to be prepared for failure cases of all shapes and sizes.

You don't know if your plan will survive contact until it's survived contact.

https://medium.com/netflix-techblog/the-netflix-simian-army-...

> Who watches the watchers?

You would need multiple watchers who watch one another. Of course, if all watchers die at the same time, you're out of luck.

This is only part of the solution though, because in order to write and test effective alerting rules, you need to repeatedly and frequently test them with either real or simulated failures, and it can get expensive.

We use Google Cloud Alerting to watch that our alerting works.

I would have used some Heartbeatservice but thats not feasible in our environment.

This is what end to end and integration testing is for.

You would set up an integration test to trigger the state that results in this alert.

haha, curious what the actual context behind that statement is. one thing id ask them is does anyone on that team understand how that system works? also, what's the reason it was broken?