Re:Scrape like the big boys
by thatwasunusual on Friday November 05, 2021 @06:30PM
(#29125418)
It sucks when this happens, but it's easily avoidable by using a caching frontend of some sort.
My favorite is Varnish,[0] which I have used with great success for _many_ web sites throughout the years. Even a web site that 10+ millions of requests per day ran from a single web server for a long time a decade-ish ago.
My favorite is Varnish,[0] which I have used with great success for _many_ web sites throughout the years. Even a web site that 10+ millions of requests per day ran from a single web server for a long time a decade-ish ago.
[0] https://varnish-cache.org/