News for hackers. Stuff that matters.
Re:Show HN: Kimono – Never write a web scraper again by randomdrake on Wednesday January 15, 2014 @05:56PM (#7066834)
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward.

I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an error that something went wrong. Well, crap. That was a waste of time. Oh well, maybe it's just me.

Alright, I'll give it another shot using the website they used in the demo. Opened up a Hacker News discussion page and started to give it a try. Immediately it was far less intelligent than the demo. Clicking on a title proceeded to select basically every link on the page. Somehow I clicked on some empty spots as well. Nothing was being intelligently selected like it was in the demo. Fine, that wasn't working tremendously well, but I wanted to at least see the final result.

Same thing: just got an error that something went wrong and it couldn't save my work.

Disappointing. I still might try it again when it works 'cause it's a great idea if they really pulled it off. So far: doesn't seem to be the case.

Re:Show HN: Kimono – Never write a web scraper again by pranade on Wednesday January 15, 2014 @06:02PM (#7066859)
Sorry you had a bad first experience. We've tested this on a lot of sites, and it works stably across a lot of different cases, but haven't solved it everywhere. Thanks for letting us know about the discussion page. We'll look into the bugs right now
Re:Show HN: Kimono – Never write a web scraper again by randomdrake on Wednesday January 15, 2014 @06:15PM (#7066921)
Having written a whole lot of crawling code throughout the years, I can totally understand how monumental of a task this is. This really does look like a cool product. Glad you're actively hunting down ways to improve the demo before asking for money :)

Feel free to drop me a line if you want any specifics about the troubles I had.

Re:Show HN: Kimono – Never write a web scraper again by faceyspacey on Wednesday January 15, 2014 @07:43PM (#7067285)
hacker news has probably been scraped too many times. that's why it's not working. it didn't work for me, but other sites worked easily.

...i will comment that the user experience for first adding elements is not what i expected. once u understand it's great overall, but i'd have some popup tooltip type guide thing going on for the first time a user uses it explaining that you should choose one element first, and then check a matching element. ...Then explain the number bubbles next to the property field. You don't get what those numbers are at first. ..Then for the property field, somehow make it more obvious that you should fill it out.

Basically you have the interface setup so any action can be done at any time, but it should be presented as a forced sequential set of steps to the end user, at least at first.

Re:Show HN: Kimono – Never write a web scraper again by pranade on Wednesday January 15, 2014 @07:46PM (#7067301)
Thanks for the suggestions on tool tips... We can also expand the tutorials section to make this more clear.
Re:Show HN: Kimono – Never write a web scraper again by ricardobeat on Wednesday January 15, 2014 @09:23PM (#7067651)
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).
Re:Show HN: Kimono – Never write a web scraper again by wilg on Thursday January 16, 2014 @12:12AM (#7068193)
I really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup?

I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible.

I can't tell you how often I click next page to find that I've taken too long and my session or whatever has expired.

Re:Show HN: Kimono – Never write a web scraper again by tinco on Thursday January 16, 2014 @07:44AM (#7069032)
No. This site isn't about UX or innovation, it's about tech start ups. It's a constant reminder that something can be successful even if it was written in a LISP dialect and has a bunch of UX misses as long as the core idea is valid and the product usable enough.

I've used a bunch of HN skins that were supposedly better designed, but none of them stuck. Apparently it's just plain unnecessary for HN to be better.

Re:Show HN: Kimono – Never write a web scraper again by elwell on Thursday January 16, 2014 @10:58AM (#7070050)
Couldn't agree more. E.g., when Digg got popular and redesigned their site everyone switched to the bare-bones Reddit.
Re:Show HN: Kimono – Never write a web scraper again by theg2 on Thursday January 16, 2014 @09:46AM (#7069618)
So because it's written in LISP it gets a free pass in other areas? That seems like a dubious claim.
Re:Show HN: Kimono – Never write a web scraper again by SkyMarshal on Thursday January 16, 2014 @04:01PM (#7072523)
No, despite it being written in an obscure language and sparse UI it has massive traction anyway. There's a lesson in there somewhere.
Re:Show HN: Kimono – Never write a web scraper again by bmelton on Friday January 17, 2014 @02:50PM (#7078008)
> There's a lesson in there somewhere

The website that lets technologists hang out with people who might give them large sums of money to see their ideas succeed doesn't have to be good, or pretty.

If the website were better, or prettier, that would not add any additional value to the previously mentioned large sums of money.

(Oh, and advice, and being able to chat with industry leaders and experts on diverse arrays of topics, etc.)

Re:Show HN: Kimono – Never write a web scraper again by Flimm on Thursday January 16, 2014 @11:13AM (#7070168)
That's not what he said.
Re:Show HN: Kimono – Never write a web scraper again by tinco on Thursday January 16, 2014 @09:58AM (#7069675)
Indubitably.
Re:Show HN: Kimono – Never write a web scraper again by precisioncoder on Thursday January 16, 2014 @08:26AM (#7069217)
I like HN because it gives me exactly what I want. A list of interesting links, ranked by what people that I truest more than news sites think of them. Oh and offers comment sections for each of them. HN is wonderful link discovery during compile time or if you need a break.
Re:Show HN: Kimono – Never write a web scraper again by LordIllidan on Thursday January 16, 2014 @10:07AM (#7069714)
I think we all like HN, otherwise we wouldn't be here. That doesn't mean it can't improve.
Re:Show HN: Kimono – Never write a web scraper again by romaniv on Thursday January 16, 2014 @10:23AM (#7069827)
If "improve" would involve turning it into AJAX-heavy app with images, CSS effects and some weird-ass scrolling interceptors, that requires me to load 200+ files for seeing the main page, I would rather live without any improvements.

I think better terms might be "clean up" or "restructure".

Re:Show HN: Kimono – Never write a web scraper again by LordIllidan on Thursday January 16, 2014 @03:06PM (#7072159)
And why do you assume that is what I meant by "improve"?
Re:Show HN: Kimono – Never write a web scraper again by Houshalter on Thursday January 16, 2014 @12:59PM (#7071167)
A minimalist interface is fine but parts of HN are simply broken or suboptimal.
Re:Show HN: Kimono – Never write a web scraper again by voltagex_ on Wednesday January 15, 2014 @09:51PM (#7067760)
A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.
Re:Show HN: Kimono – Never write a web scraper again by pranade on Wednesday January 15, 2014 @10:15PM (#7067850)
Kimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.
Re:Show HN: Kimono – Never write a web scraper again by tlarkworthy on Thursday January 16, 2014 @08:13AM (#7069162)
I developed some hot strategies for the next level ...

http://edinburghhacklab.com/2013/09/probabalistic-scraping-o...

Re:Show HN: Kimono – Never write a web scraper again by Nilzor on Friday January 17, 2014 @02:13AM (#7074680)
Have you thought of implementing a system for custom handling of edge cases? Realistically you can't handle all edge cases, so leaving the last 2% up to the user (which is a developer) would be a good idea. It'd still save him 98% of the work, but also give him the comfort of knowing it won't break even if site X is doing something weird with their HTML in the future.

I'll leave it to you to figure out how one would implement such a feature ;-)

Re:Show HN: Kimono – Never write a web scraper again by voltagex_ on Thursday January 16, 2014 @12:10AM (#7068189)
Sounds good. I've made a note of it for the next time I need to screen scrape something.
Re:Show HN: Kimono – Never write a web scraper again by ronaldx on Thursday January 16, 2014 @11:35AM (#7070361)
This is awkward, but I disagree.

HN pages are very well structured and are probably an ideal case for an automatic scraper.

If your automatic scraper doesn't work on HN, it is unlikely to work generally.

Re:Show HN: Kimono – Never write a web scraper again by roel_v on Thursday January 16, 2014 @06:03AM (#7068904)
OK so you're saying that instead of using a scraper to deal with malformed data out there (the whole reason of its existence), instead we should use a format that is better suited for machine representation? That's like saying 'yeah I've got this car here to take you to places that are very far away, too far to walk; except it doesn't work very well if you want to go far away, so you're betting off just staying at home. Or walk along time.'.
Re:Show HN: Kimono – Never write a web scraper again by ricardobeat on Thursday January 16, 2014 @08:59AM (#7069368)
Scrapers exist to take data out of HTML, malformed or not, when an API or feed is not available. Yes, ideally HTML is written in a semantic, annotated machine-parseable way; that also enables smarter search engines, better accessibility, interoperability and so on. That's one of the main reasons behind the changes to the standard made in HTML5.

A better example would be "hey I have this cool device that will improve your mileage by 100%, but it only works on cars built according to current emission standards...".

Re:Show HN: Kimono – Never write a web scraper again by pranade on Thursday January 16, 2014 @12:40PM (#7070966)
Kimono can handle many pages with malformed/ old HTML. Of course, it's still beta and there are still pages that break it, so we're improving it as we go with the help of early adopters like everyone on this thread. Of course, our goal is an ideal state it works everywhere perfectly :)
Re:Show HN: Kimono – Never write a web scraper again by easy_rider on Thursday January 16, 2014 @07:39AM (#7069017)
I'd love to argue that PG probably thinks this is a good thing. (no scraping). And for good reasons. Who is paying the bandwidth bills in the end, and for whom, you might ask. And then if those users will end up contributing to HN in any intellectual way in the end, or are just scraping content to spin in their auto-blogs filled with ads.
Re:Show HN: Kimono – Never write a web scraper again by riffraff on Thursday January 16, 2014 @09:13AM (#7069444)
hn pages are not too hard, each story is a <tr>, title is in 'tr .title' and link is in 'tr .title a'.

There are some irregularities (i.e. YC announcements without score, more button, self posts) but it's not really structurally complicated (compare reddit, which has 3 "score" fields per link)

Re:Show HN: Kimono – Never write a web scraper again by BinaryBird on Wednesday January 15, 2014 @09:11PM (#7067620)
In my case, worked for some pages and not for others. Currently I'm using Feedity: http://feedity.com for all business-centric data extraction and it has been working great (although not as flexible as kimono).
Re:Show HN: Kimono – Never write a web scraper again by sswaner on Wednesday January 15, 2014 @08:54PM (#7067558)
It worked great for me (http://www.kimonolabs.com/kimonoapp/aws-status-check). It would be helpful if you shared what you tried to do when it failed. I don't work for or with Kimono, but am curious what does and does not work.
Re:Show HN: Kimono – Never write a web scraper again by motyar on Thursday January 16, 2014 @01:19AM (#7068338)
You must check http://webscrapemaster.com/ Doing one thing well.