News for hackers. Stuff that matters.
Re:Cloudflare Crawl Endpoint by janalsncm on Tuesday March 10, 2026 @08:27PM (#47330492)
How would they know the content hasn’t changed without hitting the website?
Re:Cloudflare Crawl Endpoint by coreq on Tuesday March 10, 2026 @10:36PM (#47331204)
They wouldn't, well there's Etag and alike but it still a round trip on level 7 to the origin. However the pattern generally is to say when the content is good to in the Response headers, and cache on that duration, for an example a bitcoin pricing aggregator might say good for 60 seconds (with disclaimers on page that this isn't market data), whilst My Little Town news might say that an article is good for an hour (to allow Updates) and the homepage is good for 5 minutes to allow breaking news article to not appear too far behind.
Re:Cloudflare Crawl Endpoint by OptionOfT on Tuesday March 10, 2026 @11:16PM (#47331383)
Caching headers?

(Which, on Akamai, are by default ignored!)

Re:Cloudflare Crawl Endpoint by cortesoft on Wednesday March 11, 2026 @12:36AM (#47331780)
Keeping track of when content changes is literally the primary function of a CDN.