News for hackers. Stuff that matters.

Ask HN: Web Scraping with Node.js

Anyone out there have a cachet of good Node web scraping libs, and/or other resources? Opinions?

I have the intuition that my Google searches are a bit inadequate. Thank you!

( 6 comments | 5 points | on HN )
Re:Ask HN: Web Scraping with Node.js by iends on Monday June 20, 2016 @09:03PM (#11942812)
Re:Ask HN: Web Scraping with Node.js by tmpfs on Monday June 20, 2016 @09:12PM (#11942849)
I found https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown - a cli for validating web pages etc.
Re:Ask HN: Web Scraping with Node.js by jneumann004 on Monday June 20, 2016 @04:36PM (#11941341)
I haven't used this for web scraping, but have you tried out http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.
Re:Ask HN: Web Scraping with Node.js by paulzerkel on Monday June 20, 2016 @10:17PM (#11943064)
Here is a place to start: https://github.com/lorien/awesome-web-scraping/blob/master/j...

You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.

Re:Ask HN: Web Scraping with Node.js by pesfandiar on Monday June 20, 2016 @03:58PM (#11940948)
Should it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.
Re:Ask HN: Web Scraping with Node.js by Nilef on Monday June 20, 2016 @07:13PM (#11942343)