Re:Ask YC: Where to start with creating a distributed crawler
by krishna2 on Sunday March 30, 2008 @04:15PM
(#150379)
+10 for Python
+1 for Nutch
+1 for Hadoop
+1 for Amazon EC2
I think you have bundled two things (crawler and parser (or may be scraper)) into one term : crawler.
Beautiful soup is ok. Give html5lib a try (on google code) - but at some point you are going to have to hack the parser, but that depends on what kind of parsing you want to do.
I think you have bundled two things (crawler and parser (or may be scraper)) into one term : crawler.
Beautiful soup is ok. Give html5lib a try (on google code) - but at some point you are going to have to hack the parser, but that depends on what kind of parsing you want to do.