Show HN: Kimono – Never write a web scraper again

Re:Show HN: Kimono – Never write a web scraper again by DanBlake on Wednesday January 15, 2014 @06:52PM (#7067092)

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by pranade on Wednesday January 15, 2014 @06:55PM (#7067106)

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by chii on Wednesday January 15, 2014 @10:04PM (#7067808)

how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data?

of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by the_french on Wednesday January 15, 2014 @11:23PM (#7068071)

Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by garyjob on Thursday January 16, 2014 @12:45AM (#7068263)

The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly.

The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension.

The route of proxying the website does in fact do away with the need to install any external 3rd libraries.

This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well.

https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by davedx on Thursday January 16, 2014 @03:08AM (#7068559)

Hah, I've been working on this recently with Facebook, on a TV set-top-box. It was painful and I ended up giving up. xd_arbiter.php is the key, I think.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by billmalarky on Thursday January 16, 2014 @11:50AM (#7070496)

I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by pranade on Thursday January 16, 2014 @12:26PM (#7070828)

We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by billmalarky on Wednesday January 29, 2014 @06:53AM (#7143294)

Glad to hear it. I was just saying that I think auth should be included as a basic feature in all paid for plans.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by easy_rider on Thursday January 16, 2014 @07:42AM (#7069025)

I'm wondering how you will be able to with the numerous ways of CSRF protection implementations.

[ Parent | on HN ]

Re:Show HN: Kimono – Never write a web scraper again by automately on Thursday January 16, 2014 @02:48PM (#7072033)

Creator of Automately here, our service could definitely be something you might be interested in. While we aren't directly in the business of web scraping, we do have a powerful automation service that can accomplish those needs using simple javascript and our powerful scalable automation API.

We are accepting early access requests right now. Check us out! http://automate.ly/

[ Parent | on HN ]