News publishers limit Internet Archive access due to AI scraping concerns

Re:News publishers limit Internet Archive access due to AI s... by terminalshort on Saturday February 14, 2026 @09:21PM (#47020465)

Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?

[ Parent | on HN ]

Re:News publishers limit Internet Archive access due to AI s... by heavyset_go on Sunday February 15, 2026 @07:32AM (#47023190)

HEAD requests

[ Parent | on HN ]

Re:News publishers limit Internet Archive access due to AI s... by Findecanor on Sunday February 15, 2026 @09:16AM (#47023839)

Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?

[ Parent | on HN ]