Re:Websites and web developers mostly don't care about clien...
by snowwrestler on Saturday August 23, 2025 @08:08AM
(#44995384)
Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB).
I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.
I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.