News for hackers. Stuff that matters.
Re:Root cause analysis: significantly elevated error rates o... by laCour on Friday July 12, 2019 @01:45PM (#20422680)
"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons."

How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

Re:Root cause analysis: significantly elevated error rates o... by lethain on Friday July 12, 2019 @02:53PM (#20423332)
(Stripe infra lead here)

This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

Re:Root cause analysis: significantly elevated error rates o... by aeyes on Friday July 12, 2019 @03:50PM (#20423839)
You might want to clarify this in the post. To me it reads like you knowingly had degraded infra for days leading up to an incident which might have been preventable had you recovered this instances.
Re:Root cause analysis: significantly elevated error rates o... by lethain on Friday July 12, 2019 @05:55PM (#20424876)
Thanks for the suggestion, we’re adding a clarifying note to the report’s timeline.
Re:Root cause analysis: significantly elevated error rates o... by throwaway3489 on Friday July 12, 2019 @04:46PM (#20424343)
I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this:

"[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system."

could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead relying on containing it as a "known" vulnerable database?

The reason I ask is that I heard of ATM's still running windows XP or stuff like that. but if it's not networked could it be that that actually has a bigger uptime than anything you can do on windows 7 or 10?

what I mean is even though it is hilariously out of date to be using windows xp, still, by any measure it's had a billion device-days to expose its failure modes.

when you upgrade to the latest minor version of databases, don't you sacrifice the known bad for an unknown good?

excuse my ignorance on this subject.

Re:Root cause analysis: significantly elevated error rates o... by redis_mlc on Saturday July 13, 2019 @07:58PM (#20430859)
> could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead relying on containing it as a "known" vulnerable database?

This is a valid question.

As a database and security expert, I carefully weigh database changes. However, developers and security zealots typically charge ahead "because compliance."

Email me if you need help with that.

Re:Root cause analysis: significantly elevated error rates o... by Thorrez on Saturday July 13, 2019 @03:14AM (#20426930)
You could use that same logic to argue that they should never write any new code, just live forever on the existing code.

But customers want new features, so Stripe does changes.

Re:Root cause analysis: significantly elevated error rates o... by Jorsiem on Friday July 12, 2019 @08:31PM (#20425759)
How do you have a ATM thats not networked?
Re:Root cause analysis: significantly elevated error rates o... by throwaway3491 on Friday July 12, 2019 @09:17PM (#20425904)
Same user (sorry I guess I didn't enter my password carefully as I can't log in.)

Well I mean they're not exactly on the Internet with an IP address and no firewall, are they? (Or they would have been compromised already.)

Whatever it is, it must be separated off as an "insecure enclave".

So that's why I'm wondering about this technique. You don't just miss out on security updates, you miss performance and architecture improvements, too, if you stop upgrading.

But can that be the path toward 100% uptime? Known bad and out of date configurations, carefully maintained in a brittle known state?

Re:Root cause analysis: significantly elevated error rates o... by Operyl on Saturday July 13, 2019 @12:03AM (#20426430)
Secure .. enclave? I'm sorry but I think you're throwing buzzwords around hoping to hit a homerun here.
Re:Root cause analysis: significantly elevated error rates o... by nitrogen on Saturday July 13, 2019 @03:08AM (#20426912)
No, it's a fair question. The word "enclave" has a general meaning in English as a state surrounded entirely by another, or metaphorically a zone with some degree of isolation from its surroundings.

So the legit question is, can insecure systems (e.g. ancient mainframes) be wrapped by a security layer (WAF, etc.) to get better uptime than patching an exposed system?

Re:Root cause analysis: significantly elevated error rates o... by throwaway3491 on Saturday July 13, 2019 @03:44AM (#20426999)
yes, thank you.
Re:Root cause analysis: significantly elevated error rates o... by ashelmire on Friday July 12, 2019 @03:05PM (#20423436)
If you can think of every possible failure and create monitoring and reporting for it before it happens, then you're the best dev on the planet.
Re:Root cause analysis: significantly elevated error rates o... by sithlord on Friday July 12, 2019 @04:50PM (#20424374)
And also have the greatest bosses on the history of earth giving you unlimited time to do this.
Re:Root cause analysis: significantly elevated error rates o... by raverbashing on Saturday July 13, 2019 @04:43AM (#20427135)
And then filtering for a lot of crap and false alarms the tools and supporting infrastructure throws

I kinda lost count of how many times Nagios barfed itself and reported an error while the application was fine

Re:Root cause analysis: significantly elevated error rates o... by gtirloni on Friday July 12, 2019 @01:57PM (#20422820)
In this environment:

Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes.

having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.

Re:Root cause analysis: significantly elevated error rates o... by runevault on Friday July 12, 2019 @09:33PM (#20425960)
this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.
Re:Root cause analysis: significantly elevated error rates o... by NikolaeVarius on Friday July 12, 2019 @02:06PM (#20422916)
In many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recover

The article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.

Re:Root cause analysis: significantly elevated error rates o... by laCour on Friday July 12, 2019 @02:27PM (#20423100)
Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.
Re:Root cause analysis: significantly elevated error rates o... by kortilla on Friday July 12, 2019 @07:30PM (#20425451)
The health check said it was ok. How would they know it needed to be recovered?

The fault was the bad health check. Not the process.

Re:Root cause analysis: significantly elevated error rates o... by laCour on Friday July 12, 2019 @11:49PM (#20426391)
They only just clarified that monitoring was in place and they were reporting as healthy. See the comments above.