I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica?
That's not the case.
> If someone were to directly write to MySQL shards there would be stale data
For the sake of this discussion, you can assume this is the setup, and everything should still stand.
> I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica?
You should join us :p This is getting into the details of our cache invalidation protocol and our cache consistency model. Maybe another post on another day!
That's not the case based on our experience at FB. At-least-once delivery is a solved problem basically.
But you are absolutely right that if there's an issue in the invalidation delivery, it's possible that polaris won't receive the event as well. Polaris actually supports a separate event stream (all the way from client initiated writes) to cover this case.