News for hackers. Stuff that matters.

A comment on News publishers limit Internet Archive access due to AI scraping concerns. Parent comment

Re:News publishers limit Internet Archive access due to AI s... by iririririr on Saturday February 14, 2026 @04:48PM (#47018720)
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed.

Maybe they vibecoded the crawlers. I wish I were joking.

[ Parent | on HN ]
Re:News publishers limit Internet Archive access due to AI s... by terminalshort on Saturday February 14, 2026 @09:21PM (#47020465)
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
[ Parent | on HN ]
Re:News publishers limit Internet Archive access due to AI s... by heavyset_go on Sunday February 15, 2026 @07:32AM (#47023190)
HEAD requests
[ Parent | on HN ]
Re:News publishers limit Internet Archive access due to AI s... by Findecanor on Sunday February 15, 2026 @09:16AM (#47023839)
Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?
[ Parent | on HN ]