News for hackers. Stuff that matters.
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by adriancooney on Thursday April 28, 2016 @06:14AM (#11587649)
This is a fantastic idea and I'm really surprised nothing like this has existed before, it seems like such a no-brainer. Great work.
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by fizx on Thursday April 28, 2016 @06:55AM (#11587815)
https://github.com/fizx/parsley/wiki looks pretty similar.

Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt denied and/or blocked.

We briefly tried, and supported a wiki of json extraction scripts at parselets.org, but it went nowhere after a few months.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by geuis on Thursday April 28, 2016 @07:36AM (#11587994)
I built something almost identical in 2011. It really doesn't have as much utility in practice as you think initially. CSS selectors are an interesting idea for extracting data from pages, but it's extremely fragile. You have to either parse the page's raw html using something like jsdom, or you run it through a headless browser like Phantom. In the first case, it completely fails for any modern SPA (angular, react, etc). In the second case, phantom is painfully slow and difficult to interact with, and often doesn't run/render an SPA as a regular browser does.

You can write tests around whether your selectors are returning data, but even simple refactors from a dev team quickly break your selector profiles multiple times a week or month.

Just wasn't worth the hassle.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by mickael-kerjean on Friday April 29, 2016 @07:42AM (#11595021)
There is some solutions to run a SPA in real browser even in a headless environment

The trick is to emulate x11 with xvfb and control it with selenium web driver.

Phantom isn't the only choice, just the one most people talk about

As for non js heavy website, it's fairly trivial to find a library that will parse the dom for you, pretty every language have one

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by NicoJuicy on Thursday April 28, 2016 @10:27AM (#11589069)
Done it also, to scrape HN in cli :p
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by LoSboccacc on Thursday April 28, 2016 @07:48AM (#11588057)
done it as well. at time specialized to organize web comics (it was way before google reader times).

real issue is that popularity will get you blocked fairly quickly. see also: yahoo pipes.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by moeamaya on Thursday April 28, 2016 @09:19AM (#11588555)
There was a YC company a few years back that got acquired by Palantir in February that did something very similar. https://www.kimonolabs.com/
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by jancurn on Thursday April 28, 2016 @07:58AM (#11588123)
Well, https://www.apifier.com does essentially the same thing, plus it supports JavaScript, can crawl through the whole website etc.

Disclaimer: I'm a cofounder there

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by novaleaf on Thursday April 28, 2016 @11:21AM (#11589503)
so does https://PhantomJsCloud.com but single pages, no site crawling.

Disclaimer: I'm the founder there ;)

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by fiatjaf on Thursday April 28, 2016 @09:03AM (#11588469)
Apifier looks actually awesome.
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by xaduha on Thursday April 28, 2016 @06:22AM (#11587670)
Using a 3rd party site to query HTML (which you should be able to do yourself, plenty of tools for that) isn't a fantastic idea.

This one, for example http://www.videlibri.de/xidel.html#examples

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by martinvol on Thursday April 28, 2016 @06:23AM (#11587674)
The code is on Github, you can use this as a library, not as a 3rd party saas.
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by lorenzhs on Thursday April 28, 2016 @08:03AM (#11588154)
kimono labs used to do something similar, but shut down recently. They had a nice clicky pointy interface that allowed you to build the selectors by clicking on elements, with an immediate preview. They also handled things like pagination etc.
Re:Show HN: Jam API, turn any site into a JSON api using CSS... by mappy on Thursday April 28, 2016 @07:13AM (#11587882)
> I'm really surprised nothing like this has existed before

But how would you monetize it?

Unlike an RSS feed, you really don't know how the JSON response would be used, so you can't inject ads into it.

And if you charge for it, wouldn't people assume it would continue to work, but site "scrapers", regardless of how they are configured, are likely to break, so it would be tougher having customers pay for something that could break at any time leaving them having to figure out if its the service that's changed/broken or the page that's changed.

Don't get me wrong- some great businesses have been/are based on "scraping" in one way or another. However, as cool as this is, it's just another way to "scrape". If the person hosting the page would provide an API or JSON view instead, you'd be loads better off.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by nsgi on Thursday April 28, 2016 @08:08AM (#11588178)
Freemium, professional support, expanding it into an abstraction layer above the APIs for multiple services, selling a version that larger companies can run on their own servers which they might need for data security...

In any case, not everything has to be monetised.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by g00gler on Thursday April 28, 2016 @09:54AM (#11588841)
>However, as cool as this is, it's just another way to "scrape"

Isn't that the point? The demo seems like it'd be a lot easier, less verbose, and probably less brittle, than using cUrl/xpaths or otherwise parsing that HTML yourself.

Re:Show HN: Jam API, turn any site into a JSON api using CSS... by phsource on Thursday April 28, 2016 @12:59PM (#11590280)
We launched WrapAPI (https://wrapapi.com/) a few weeks ago with the same functionality, but a bit more complex and powerful process to get set up. You can not only specify CSS selectors yourself but define them point and click.

The barrier for starting with JamAPI is impressively low, though! Kudos on the developer-friendly user interface.