A Sochi Olympics API

Re:A Sochi Olympics API by mbesto on Tuesday February 18, 2014 @02:34PM (#7260495)

Serious question - have you look very deep into the legality of what you're doing? I have no doubt in the world there is huge value in your service but generally speaking your website violates the most basic terms of use of pretty much any site that exists. That being said - you (and pretty much every other web scraper in the world) enters into this grey area of "binding contract" in the terms of use. Personally I'd be very scared about your service being able to scale for two fundamental reasons (1) the more attention you attract as you scale, the more legislation will be targeted at you and (2) people who want to protect their data will create services to block your efforts.

note - IANAL and I say this not to scare you, but hopefully to help.

[ Parent | on HN ]

Re:A Sochi Olympics API by meritt on Tuesday February 18, 2014 @04:32PM (#7261242)

There are numerous web-scraping tools out there, many are easy enough to use for non-programmers (Connotate and Mozenda for instance) so, while you have a point, precedent shows that web scraping is prominent and not going away anytime soon. I applaud innovation in this area.

Your #2 is simply not plausible unless Kimono carefully follows robots.txt and consistent user-agent, which is rarely the case for a competent web-scraping platform.

Legality is also a strong word. Web scraping is rarely illegal in the sense that you're committing a crime. A company could definitely issue a C&D and potentially go after you in a civil lawsuit but it's very infrequent (sadly not so much anymore with all the bullshit CFAA suits against "hackers" lately) that a government entity would pursue someone for scraping data.

The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping.

[ Parent | on HN ]

Re:A Sochi Olympics API by mbesto on Tuesday February 18, 2014 @05:03PM (#7261332)

Precedent is also a strong word. There are numerous people on the road who go over the speed limit. It's prominent and not going away anytime soon. This is not a cause for precedence.

Precedence implies that a legal case take place in order to dictate the law. AFAIK, between your unsubstantiated comment and my general knowledge of this area, the reason no one has provided a scraper service at scale isn't because the technology or execution wasn't right, but rather that it's simply illegal to do so.

> The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping.

That's simply not true. Here's an example:

http://www.yelp.com/static?country_=US&p=tos

"By accessing or using the Site, you are agreeing to these Terms and concluding a legally binding contract with Yelp Inc., a Delaware corporation headquartered in San Francisco, California ("Yelp"). Do not access or use the Site if you are unwilling or unable to be bound by the Terms."

"Use any robot, spider, site search/retrieval application, or other automated device, process or means to access, retrieve, scrape, or index any portion of the Site or any Site Content;"

My overall point? Kimonolabs is slowly painting a target on their back and it's only a matter of time before they'll have to "lawyer up".

[ Parent | on HN ]