News for hackers. Stuff that matters.
Re:Palantir acquires Kimono by minimaxir on Monday February 15, 2016 @04:35PM (#11106204)
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.

I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.

Re:Palantir acquires Kimono by dwynings on Monday February 15, 2016 @05:01PM (#11106374)
Hi Max,

All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...

If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/

Re:Palantir acquires Kimono by mistermann on Monday February 15, 2016 @04:47PM (#11106287)
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:

http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...

Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?

Re:Palantir acquires Kimono by dwynings on Monday February 15, 2016 @05:06PM (#11106412)
Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/

If you'd like me to set you up with a demo, feel free to email me: [email protected]

Re:Palantir acquires Kimono by Bammybums on Tuesday February 16, 2016 @06:04AM (#11109095)
if you run HN front page through import.io's magic, it works pretty well: https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...