Re:Show HN: Jam API, turn any site into a JSON api using CSS...
by mickael-kerjean on Friday April 29, 2016 @07:42AM
(#11595021)
There is some solutions to run a SPA in real browser even in a headless environment
The trick is to emulate x11 with xvfb and control it with selenium web driver.
Phantom isn't the only choice, just the one most people talk about
As for non js heavy website, it's fairly trivial to find a library that will parse the dom for you, pretty every language have one
Re:Show HN: Jam API, turn any site into a JSON api using CSS...
by NicoJuicy on Thursday April 28, 2016 @10:27AM
(#11589069)
Done it also, to scrape HN in cli :p
Re:Show HN: Jam API, turn any site into a JSON api using CSS...
by LoSboccacc on Thursday April 28, 2016 @07:48AM
(#11588057)
done it as well. at time specialized to organize web comics (it was way before google reader times).
real issue is that popularity will get you blocked fairly quickly. see also: yahoo pipes.
You can write tests around whether your selectors are returning data, but even simple refactors from a dev team quickly break your selector profiles multiple times a week or month.
Just wasn't worth the hassle.