Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How to write a crawler.
Multi-threaded. Built-in page delay (200ms default). Does HTTPS, headers, POST, cookies, follows robots.txt .
I prefer it to nutch for small to medium sized jobs.
https://code.google.com/p/crawler4j/
There are no comments on this comment yet.
Multi-threaded. Built-in page delay (200ms default). Does HTTPS, headers, POST, cookies, follows robots.txt .
I prefer it to nutch for small to medium sized jobs.
https://code.google.com/p/crawler4j/