News for hackers. Stuff that matters.

A comment on Show HN: Scrape the web by injecting JavaScript into web pages.

Re:Show HN: Scrape the web by injecting JavaScript into web ... by ivanca on Tuesday December 23, 2014 @02:16PM (#8789688)
This is an interested idea, but what I would pay for is a GUI for scrapping, for example: clicking an element, seeing what classes it got, and be able to click each class to see if it matches the elements I need. And then generate the code for those actions I did in the GUI, maybe even be able to do both (modify the JS and still work with the GUI when required, yeah, is hard but possible)

The other major feature I need is paths of execution, for example if there are two possible pages after certain step (think if-else) I want those views visualized as interconnected nodes.

[ Parent | on HN ]
Re:Show HN: Scrape the web by injecting JavaScript into web ... by juddernaught on Tuesday December 23, 2014 @04:31PM (#8790338)
If you are looking for a GUI for scraping, check out kimono labs. It's a GUI tool for scraping, and requires no code to set up and can find all similar elements from the one you click on. It supports pagination and other types of scrapes too.
[ Parent | on HN ]
Re:Show HN: Scrape the web by injecting JavaScript into web ... by aaronem on Tuesday December 23, 2014 @02:36PM (#8789785)
Selenium could do that for you, and there's an IDE available as a Firefox plugin; branching might be tricky and you'd need to build some tooling around it in general, but it might give you a place to start.
[ Parent | on HN ]