News for hackers. Stuff that matters.
Re:Root cause analysis: significantly elevated error rates o... by gtirloni on Friday July 12, 2019 @01:57PM (#20422820)
In this environment:

Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes.

having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.

Re:Root cause analysis: significantly elevated error rates o... by runevault on Friday July 12, 2019 @09:33PM (#20425960)
this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.