News for hackers. Stuff that matters.
Re:Ask HN: How you scrape websites? by remx on Sunday May 14, 2017 @10:02AM (#14335649)
Greasemonkey[0] is fairly handy for scraping. You can even include remote resources like jQuery if you're targeting specific DOM elements in the page. For storing the data, you can just use LocalStorage (for example if you're paginating)

[0] https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/

Re:Ask HN: How you scrape websites? by stefanpie on Sunday May 14, 2017 @11:11PM (#14338888)
99% of the time I can easily thow together a simple Python script with the BeautifulSoup and requests libraries to do any kind of basic scraping, form text to images or to API endpoints. And most of the time if I need a to store the data nicely I can either use the built-in csv or json modules or use SQLite database as a single file (which I believe is also part of the standard library)
Re:Ask HN: How you scrape websites? by brandonlipman on Friday May 12, 2017 @11:54PM (#14329466)
So many ways to do this. What are you trying to scrape? If its light weight, you could use a Chrome plugin called Data Miner. Right now I am using Beautiful Soup for most of my scrapping and love it. Python is incredibly easy to learn and powerful.
Re:Ask HN: How you scrape websites? by motyar on Monday May 15, 2017 @11:00AM (#14342077)
Re:Ask HN: How you scrape websites? by egfx on Tuesday May 16, 2017 @08:31PM (#14354594)
Try with another. The qkast web clipper http://qkast.com
Re:Ask HN: How you scrape websites? by jetti on Thursday May 18, 2017 @01:09PM (#14368980)
It has been awhile since I last did this but I used HTMLAgilityPack for .NET.
Re:Ask HN: How you scrape websites? by limeblack on Sunday May 14, 2017 @01:17PM (#14336440)
I've used Phatom.js before. It really depends on the site. Some sites need full browsers running which you could use selenium.
Re:Ask HN: How you scrape websites? by Philomath on Saturday May 13, 2017 @04:05AM (#14329981)
If you like JavaScript you may give Nightmare.js a try. It's so simple yet so powerful. I've used it quite a lot and never had a complaint.
Re:Ask HN: How you scrape websites? by hawkweed on Monday May 15, 2017 @08:59AM (#14341180)
JSoup[0] works fine for me.

[0] https://jsoup.org/

Re:Ask HN: How you scrape websites? by curiousgal on Friday May 12, 2017 @09:40PM (#14329184)
Scrapy
Re:Ask HN: How you scrape websites? by espiii on Monday May 15, 2017 @02:07PM (#14343689)
This tool felt like it descended from the gods of software. It's a brilliant tool.
Re:Ask HN: How you scrape websites? by dhruvkar on Sunday May 14, 2017 @03:21PM (#14337097)
Depending on how complicated the site its either

Selenium with Chrome running in a virtual display.

Python requests with Beautifulsoup

Re:Ask HN: How you scrape websites? by Madeindjs on Monday May 15, 2017 @04:23PM (#14344801)
Ruby & Nokogiri gem. You can also use Anemone gem to crawl the complete website.
Re:Ask HN: How you scrape websites? by tjalfi on Saturday May 13, 2017 @09:11PM (#14333755)
Re:Ask HN: How you scrape websites? by bbcbasic on Tuesday May 16, 2017 @06:33AM (#14348576)
| grep ... | sed ... | uniq ... | etc ...
Re:Ask HN: How you scrape websites? by gerenuk on Saturday May 13, 2017 @08:27AM (#14330614)
Celery with gevent/process.