Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How to write a crawler. Parent comment
It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.
There are no comments on this comment yet.
It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.