Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on 80 legs: Web Crawler as a Service.
What User-Agent do you use?
Do you crawl non-textual resources?
Do you save all headers from the crawled responses?
Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?
If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?
Do you save the exact time of the request (not trusting the returned 'Date' header)?
There are no comments on this comment yet.
What User-Agent do you use?
Do you crawl non-textual resources?
Do you save all headers from the crawled responses?
Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?
If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?
Do you save the exact time of the request (not trusting the returned 'Date' header)?