It's important never to
require a cache hit - even
as a soft requirement.
Evictions can happen at
inconvenient times - such
as when some other part of
the system is under load -
and there mustn't be any
negative consequences to a
cache miss
If the cache does empty unexpectedly and that is going to cause your infrastructure to die in a cascade, probably some kind of fallback will be needed.It's an asynchronous read model or a projection and it's now just part of your application.
For instance if I look up a value and pass it to three pure functions to ask about it, versus having the 3 functions look up the value and try to share a cache entry.
This is doubly wrong because the final question may see a cache eviction and answer a different question due to changes in the data. Now I may have a situation where True:False:True was not a corner case we cover because it's logically impossible, but it's happening in the code because the last one changed from False to True due to concurrent read and write.
The latter is also harder to test, because it requires mocks instead of manufactured inputs. And at this point the code is using the cache as global, mutable state, instead of semi-local state. Three strikes, you're out.
It's a subtle word change, but can communicate the slightly difference architecture and break the reader away from an assumption.
If your site comes crashing down because the cache went down, it's not a cache anymore and you should use a different mindset in approaching it.
The answer varies depending on a whole lot of relevant numbers that can vary widely, but an example of an answer I have in one of my systems is that I store all the cache queries more persistently than the cache answers. This particular system gets hammered with requests that are mostly the same as last time at certain times but spends a lot of time idle. The numbers work out that if I'm trying to compute the cache during one of my crunches, I need a huge amount of resources for just a couple of minutes, but if I restart the service between these crunches, there's plenty of time to recompute the entire cache before it is necessary with no additional resources. But that's just one particular example of not really being a "cache" anymore suitable in my very particular case.
That guarantees that the cache remains hot for records that clients are requesting, but lets unused records expire.
A two second response with a 10 second TTL could see 20% of the queries all trying to refresh the same cache at once. With a single server you can use promise caching to dedupe such things, with a cluster that's more difficult to avoid.
Background processes are good for spiky traffic, and with reddit and HN around some of us at least have been trained that some information on a page is purely advisory. Do I have 22 upvotes or 28? Give it a couple minutes and try again.
But what that work does is make you look at what the cost is of certain information, and some people are not comfortable with that. With caching you can delude yourself that you're getting a bargain on all of this information overload, even though the worst case scenarios and statistical clumping say you're off your gourd.
Then you could have the cache serve stale content until it's updated, and/or only allow one upstream connection to refresh the cache.
But yeah, unless you allow stale responses or refresh the cache periodically as you suggest, you would still have requests which takes 10 second to complete, but you would at least not crush the upstream service.
Also works well to combine that with stale-while-revalidate. Yahoo had some squid patches with nifty ways to have a response invalidate other cached urls, but I don't think those made it to the outside world (at least I can't find them now).
We used that technique in Google's DNS cache to fix what had been a monumental herd effect every 30 seconds when the RRs for outlook-com.olc.protection.outlook.com expired.
In this case, per-request latency wasn't as important as total throughput of the system over time. And the individual requests were small, so building up a queue of unhandled requests wasn't a problem.
I have no idea if this is considered a "good" solution, but it worked well enough for me at the time.
> you have a fleet of web nodes going at full bore, and the cache expires. If the data takes 10 seconds to compute, then for the whole next 10 seconds, every new request will find the data missing from the cache since it has expired and not yet been replaced, and so each of those requests will take on the expensive and redundant work of repopulating that data.
Wait, is the problem that as soon as the cache entry expires everyone requests the new data all at once, overwhelming the backend ("thundering herd"), or is it that requests follow whatever the normal pattern is but the backend doesn't know how to queue them?
The naive approach is to follow the normal pattern: use the cached version of it's there, and otherwise compute and stick it in the cache with a TTL. The problem arises when you have a fleet of web nodes going at full bore, and the cache expires. If the data takes 10 seconds to compute, then for the whole next 10 seconds, every new request will find the data missing from the cache since it has expired and not yet been replaced, and so each of those requests will take on the expensive and redundant work of repopulating that data.
It can be a performance blip at best, or a total disaster at worst, causing cascading failures and downtime.
One relatively easy solution here is to just have a separate worker process that runs in an interval and preemptively freshens the data in the cache. In this case you can set with no TTL, and web front ends just use whatever they find. You also need to deal gracefully with the rare case where they may find nothing, like in the case of a cold cache boot.