News for hackers. Stuff that matters.

A comment on APIs Are Forever, Wait No...They Can Go Away at Any Time.

Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by hartleybrody on Friday April 20, 2012 @09:50PM (#3870585)
This is one of the best arguments for scraping data, if you can manage it. Generally businesses invest much more time and resources into maintaining an up-to-date, usable website than they would on an API, especially if they don't use a service-oriented architecture internally (ie, consume their own API)
[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by rollypolly on Friday April 20, 2012 @10:04PM (#3870609)
What type of business can afford being down every time the site they're scrapping is tweaked?
[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by hartleybrody on Saturday April 21, 2012 @10:19PM (#3873699)
How often do websites really change the layout of their DOM? Sure, some sites that are actively developed (ie social networks) but for a lot of business websites where information doesn't change that often, like restaurants, reviews or clothing, websites are pretty consistent.

Plus, building a parser isn't always that much more work than trying to build a client for a convoluted API.

It's obviously case-by-case, but I think a lot of people dismiss scraping outright when there are a lot of useful applications.

[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by tantalor on Saturday April 21, 2012 @02:24AM (#3871111)
1. Any business that scrapes many sites.

2. Who says it goes down? Might just have stale or incorrect data before the tweak is patched.

[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by sad_panda on Friday April 20, 2012 @10:54PM (#3870713)
OMG, what?

Scraping data for offline processing, but certainly should not be for user-facing traffic. Cosmetic tweaks can blow up the scraper's parser, UI technologies are built on a taller stack and are prone to higher latencies and points of failure, yada yada.

[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by puxedo on Saturday April 21, 2012 @02:37AM (#3871131)
Indeed. Very risky. Scraping is not really a sustainable model when you need predictability. Too many variables.
[ Parent | on HN ]
Re:APIs Are Forever, Wait No...They Can Go Away at Any Time by tantalor on Saturday April 21, 2012 @02:30AM (#3871118)
An example comes to mind:

The Yelp API doesn't expose a given user's public bookmarks, reviews, etc., but they can be easily scraped from the user-facing website.

http://www.yelp.com/developers/documentation/v2/overview

[ Parent | on HN ]