Re:Build a crawler to crawl million pages with only one mach...
by Xeoncross on Tuesday February 28, 2017 @03:09PM
(#13757387)
Javascript is the language to build a smaller crawler in because web pages run Javascript and you crawler needs to also. A C++ or Golang crawler will be a little faster and use a lot less memory - but you have to compile in webkit and do a bunch of hacky stuff to run the pages Javascript.
On the other hand, if you are building a massive crawler you want to split the crawling and parsing into two separate functions and do the network I/O in golang/c and do the parsing with a Javascript headless browser like phantom.
I don't really see any reason to use python unless you haven't learned golang (the world's biggest crawler's own language).
On the other hand, if you are building a massive crawler you want to split the crawling and parsing into two separate functions and do the network I/O in golang/c and do the parsing with a Javascript headless browser like phantom.
I don't really see any reason to use python unless you haven't learned golang (the world's biggest crawler's own language).