Sorry if it sounds like silly questions, I was genuinely curious to learn about the strategy since the blog started with "When it comes to cache invalidation, we believe we now have an effective solution to bridge the gap between theory and practice"
> what about many-to-many where one fetch returns inconsistent results Are you referring to the case when cache stores a complicated result of a function that fetches data from many sources, and how to figure out when to invalidate? If so, I am sorry but that's not the cache invalidation I was referring to. (https://news.ycombinator.com/item?id=31676102)
> What about data queried while there's a transaction rollback.. when is it updated in the cache
This might be easier to answer. I assume if you run the database with isolation level higher than read-committed, you will be fine.
Could this delay cause a wrong reporting? Since the source of truth (the DB) could already received multiple mutate events of the data Polaris trying to verify? How do you handle this case?
Yeah I have learned about some people have different perceptions of what cache invalidation means (I have a narrower definition, a subset of what some people think as cache invalidation). I will not die on this hill. I am happy to rename/rephrase/anything to be helpful.
> it doesnt sound like Polaris would handle a network partition well ...
Good question. And you are right. My thought on this is that over time, polaris would still catch those unique issues happened at network partition or whatnot. Polaris doesn't promise have 100% coverage all the time. However, over time, if there are flaws in the system, Polaris should help surface it. And once it did (even with just one inconsistent sample), it should be very actionable, and we can find out why it happened, and fix the underlying cause.
> I also feel like the consistency tracing is assuming that only one fill and invalidate is occurring at a time (when in practice, there may be multiple fills and invalidates occurring in parallel on the same data)... and that those calls will arrive in order. If they arrive out of order, it doesnt sound like it would catch that..
It doesn't make that assumption actually. Distributed systems are state machines. Consistency tracing essentially logs (it doesn't do much detection) state transitions. So when polaris detects an anomaly, we have all the information to help us diagnose. And you are right that fills and invalidations can happen in parallel and it's fine. E.g. if fill happens before invalidation, we can always track state mutations caused by the invalidation. If invalidation is the culprit (the earlier fill can't be because it happened earlier), we would have a log for it.