News for hackers. Stuff that matters.
Re:Sieve is simpler than LRU by vidarh on Wednesday January 03, 2024 @01:41PM (#38857876)
Whether that is "how it should work" and whether your proposed fix is better or worse for a given workflow depends on a lot of factors.

You're right that putting the insert in a fixed location will mean the time a new element has before it must either be visited or get evicted will vary quite a bit, but it's not obvious that it will always be better to "fix" that.

E.g. consider the typical "generation gap" where the set of objects are bimodal and either likely to be frequently used or be garbage right away. Depending on the odds that a new object falls in either category, it might be better for the insert to be closer to the hand than further away (or it might be opposite). That is, you might want the chance for a new entry to survive to be fairly low.

If the best bet is to have the new entry on average an inconvenient distance from the hand, it might well be the variability is better than either moving the insert to the hand or a more convoluted scheme to keep them at a fixed distance.

I agree it'd be more predictable and/or easier to reason about if it's a fixed distance, and if the fill size of the cache is the right distance then certainly inserting it at the hand works great.

But really, I think you should measure for your specific workflow. It'd be interesting to see this option and some other ways of separating them tested, though.

Another thing I think would be interesting to test is whether (it wasn't an accident that I hinted at generational garbage collection above) it'd be worth keeping a second pointer that you move past objects that have survived "long enough" in the cache, and most of the time move the hand back to that pointer instead of "all the way" so you don't spend as much time iterating over objects that won't get evicted anyway. If the number of cache entries is small, probably not worth it, but for a large cache with a decently large proportion of persistently hot objects, maybe (but then you could just also outright move objects to a second cache, and that might well be simpler; again something to measure on actual workloads).

I'm assuming this isn't anything new to people who follow the field more closely in any case.

Re:Sieve is simpler than LRU by kazinator on Wednesday January 03, 2024 @08:06PM (#38861891)
"Your specific workflow" is a myth. A cache designed for "your specific workflow" will be subject to completely unforseen workflows when put into production.
Re:Sieve is simpler than LRU by vidarh on Wednesday January 03, 2024 @11:17PM (#38862932)
While that's something to keep in mind, that's a reason for measuring tue effects and choosing accordingly, not a reason not to.

Otherwise on that basis you shouldn't put a cache in at all as there will be workflows where the hit rate is so low the cache has a negative effect, however unlikely.

If you don't know e.g. the rough performance profile of your requests, and the typical distribution of them, you likely have far bigger things you should be worrying about.