Re:Ask YC: Where to start with creating a distributed crawler
by groovyone on Sunday March 30, 2008 @04:33PM
(#150396)
Hi there. I think I've decided we need to build our own, mainly as what we're wanting to do is quite specific - monitoring of sites for keywords and really we should understand,to a degree, this technology. Happy to share if you end up heading down the Python route. If there is anyone else on here who is doing crawling/data mining then maybe we could share ideas, help each other somehow :) My email is in my profile
[ Parent
| on HN ]
Re:Ask YC: Where to start with creating a distributed crawler
by Readmore on Monday March 31, 2008 @12:21PM
(#151169)
Have you seen this article yet? Looks like it would be useful to you. http://blog.ianbicking.org/2008/03/30/python-html-parser-per...