News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by bravura on Friday August 10, 2012 @09:07PM (#4369038)
To build a smart spider, you can have a classifier that determines if you should crawl the outlinks of the page.

This classifier can be trained using logistic regression.

This is slightly tricky, but effective. I've been meaning to write in more depth about this topic (smart crawling).

[edit: I'm stepping out so I can't write more right now. You can email me if you have any questions about this.]

There are no comments on this comment yet.