News for hackers. Stuff that matters.
Re:Ask YC: Where to start with creating a distributed crawler by groovyone on Sunday March 30, 2008 @04:33PM (#150396)
Hi there. I think I've decided we need to build our own, mainly as what we're wanting to do is quite specific - monitoring of sites for keywords and really we should understand,to a degree, this technology. Happy to share if you end up heading down the Python route. If there is anyone else on here who is doing crawling/data mining then maybe we could share ideas, help each other somehow :) My email is in my profile
Re:Ask YC: Where to start with creating a distributed crawler by Readmore on Monday March 31, 2008 @12:21PM (#151169)
Have you seen this article yet? Looks like it would be useful to you. http://blog.ianbicking.org/2008/03/30/python-html-parser-per...