Here's a bunch of versions of variable "X":
X@Alice-V1, X@Alice-V2, X@Bob-V1, X@Alice-V2/Bob-V2, X@Carol-V1
Which version of "X" should you read?
EDIT: Alice, Bob, and Carol are three different servers. What has happened here is that Alice and Bob are closer together, so their updates are faster between each other (synchronizing the writes). Carol is slower for some reason (bad connection?), and is going to update off of old data.
In this case, the two valid caches are X@Alice-V2/Bob-V2, and X@Carol-V1. The other cached data is old and can be discarded.
Things get complicated when you have not only reads (such as in your simplified question), but also writes that are cached (such as what happens in practice).
Plus what about read workloads, like “pull message from queue and send it as a push notification to user Y”? I guess it’s fine if the push is re-delivered due to stale cache?
As long as the client submits transactions with “I calculated this based off of X@v4” then the database can reject if there is a newer version of X.
The client can then replay the transactions against the cache and by then there will be a X@v5.
With this scheme you can track the transaction replay rate instead of having to build a new system to read-back the cache for consistency.
To get the traceability on which cache node is stale, the client can forward a header from the cache node that identifies it. Then using the same transaction rejection rate ops has visibility on which cache node isn’t being updated.
No cache invalidation needed. Always just cache forever X@version, it’s just that the cache allows an unstable query for GET X.