Internet Archaeology: Scraping time series data from Archive.org

Re:Internet Archaeology: Scraping time series data from Arch... by deferredposts on Wednesday April 05, 2017 @12:16PM (#14043182)

So what is the policy of The Internet Archive on this level of scraping? Do they have a rate limit in place?

[ Parent | on HN ]

Re:Internet Archaeology: Scraping time series data from Arch... by foob on Wednesday April 05, 2017 @12:57PM (#14043524)

Yes, they start sending 429 (Too Many Requests) responses if you don't use appropriate delays. They also provide a public API [0] which I believe is intended for automated requests of this type (as opposed to crawling the Wayback Machine website directly).

[0] - https://archive.org/help/wayback_api.php

[ Parent | on HN ]