Re:Cache made consistent: Meta’s cache invalidation solution
by uvdn7 on Thursday June 09, 2022 @08:33AM
(#31680383)
My experience is that this is easier said than done. But I agree with you that in theory, one can inject failure and run tests with all possible combinations, and have a repeatable test framework (think about foundationdb’s test framework) to help root cause test failures. Sqlite has very impressive reliability; its test framework is not open sourced. It also benefits from user bug reports, as it’s literally the widest deployed db. Bugs/flaws happen.
Having better tests and better observability in production are also not mutually exclusive.
Can't we assume they tested it before deploying to production by introducing all the weird stuff that might happen in a distributed system like partitions, high latency, so they aren't relying only on "it seems to work in production"?