Re:Scraping Web Pages With jQuery, Node.js and Jsdom
by baudehlo on Thursday March 08, 2012 @11:59AM
(#3680490)
"curl" can scrape sites like that. Just watch the Chrome "Network" window and see what requests get made. Then use that. Scraping doesn't need a browser.
[ Parent
| on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom
by MatthewPhillips on Thursday March 08, 2012 @12:09PM
(#3680539)
Of course curl can. And curl was the best option 15, even 10 years ago. But to scrape a complicated site with curl (or any other non-browser scrapping utility) you wind up basically rewriting the application you're attempting to scrape.
Browsers sole purpose is to run web apps. It doesn't make sense to use anything else.
Re:Scraping Web Pages With jQuery, Node.js and Jsdom
by baudehlo on Thursday March 08, 2012 @12:41PM
(#3680707)
At the low end of the scale I agree. But once you start scaling it's very difficult to run more than around 40 concurrent sessions with a browser-based scraper.
With a pure Nodejs scraper we can run over 1000 parallel sessions per CPU.