Re:Cache made consistent: Meta’s cache invalidation solution
by uvdn7 on Wednesday June 08, 2022 @05:08PM
(#31674199)
I agree with everything you said.
> Having a cache whose design allows it to be stale but only very rarely seems like a bad idea.
And that's exactly why we first brought this number from 6 9's to more than 10 9's; and we are not done yet. Also the cache is not "designed" to be inconsistent. It's like Paxos is designed to solve the consensus problem, but when implemented, most Paxos implementations do not work properly.
Those are the kind of rates that lead to very hard to find bugs. You won't properly write code in downstream systems to properly handle that failure case, and suddenly someone will get paged at 3am because it has just happened years after the code was deployed and nobody really understands how this corner case suddenly happened!
When I'm building systems, every happy codepath should either happen frequently or never.
Having a cache whose design allows it to be stale but only very rarely seems like a bad idea. If I was building something on top of this cache I would have a layer on top to force staleness for 1 in 1000 requests just to force the application built on top to be designed to handle that properly.