throwawayscrapd

And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.

I mean given that you've had this problem repeatedly, I'd call it "past-proofing", but I suppose you know your codebase better than I do.

Did you ever consider refactoring the code so that you don't have to do shotgun surgery every time you make this kind of change?

I work on the kind of big online scientific database that this article is about.

100% of our data is available from a clearly marked "Download" page.

We still have scraper bots running through the whole site constantly.

We are not "golden".