News for hackers. Stuff that matters.
Re:Palantir acquires Kimono by minimaxir on Monday February 15, 2016 @03:47PM (#11105822)
As someone who used Kimono a few times before the closure...this announcement does not surprise me at all.

The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.

Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.

Re:Palantir acquires Kimono by michaelmior on Monday February 15, 2016 @03:57PM (#11105895)
"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
Re:Palantir acquires Kimono by x1024 on Monday February 15, 2016 @04:04PM (#11105951)
It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity:

An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.

What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable.

The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.

Re:Palantir acquires Kimono by trhway on Monday February 15, 2016 @04:33PM (#11106187)
> "non-programmers can do it. No code required, etc. etc."

translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...

Re:Palantir acquires Kimono by derefr on Monday February 15, 2016 @05:14PM (#11106456)
We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems."

You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.

Re:Palantir acquires Kimono by x1024 on Monday February 15, 2016 @07:12PM (#11107028)
I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
Re:Palantir acquires Kimono by minimaxir on Monday February 15, 2016 @04:02PM (#11105941)
Laziness is a secret form of technical debt.

For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.

Re:Palantir acquires Kimono by michaelmior on Monday February 15, 2016 @04:14PM (#11106035)
While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
Re:Palantir acquires Kimono by minimaxir on Monday February 15, 2016 @04:26PM (#11106139)
See x1024's comment above. (best case scenario is that it's not significantly faster)
Re:Palantir acquires Kimono by mrdrozdov on Monday February 15, 2016 @04:17PM (#11106070)
Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...
Re:Palantir acquires Kimono by minimaxir on Monday February 15, 2016 @04:35PM (#11106204)
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.

I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.

Re:Palantir acquires Kimono by dwynings on Monday February 15, 2016 @05:01PM (#11106374)
Hi Max,

All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...

If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/

Re:Palantir acquires Kimono by mistermann on Monday February 15, 2016 @04:47PM (#11106287)
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:

http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...

Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?

Re:Palantir acquires Kimono by dwynings on Monday February 15, 2016 @05:06PM (#11106412)
Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/

If you'd like me to set you up with a demo, feel free to email me: [email protected]

Re:Palantir acquires Kimono by Bammybums on Tuesday February 16, 2016 @06:04AM (#11109095)
if you run HN front page through import.io's magic, it works pretty well: https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...
Re:Palantir acquires Kimono by w1ntermute on Monday February 15, 2016 @03:48PM (#11105829)