News for hackers. Stuff that matters.
Re:80 legs: Web Crawler as a Service by gojomo on Wednesday April 08, 2009 @03:27PM (#553276)
Very interesting service! A number of questions...

What User-Agent do you use?

Do you crawl non-textual resources?

Do you save all headers from the crawled responses?

Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?

If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?

Do you save the exact time of the request (not trusting the returned 'Date' header)?

There are no comments on this comment yet.