Re:Congrats! Web scraping is legal! (US precedent)
by magduf on Wednesday January 29, 2020 @10:36AM
(#22181638)
I really don't understand why anyone would bother writing and using a web scraper when an API exists. Does the API not provide all the same data/functions as the website? Scrapers are a big PITA compared to just using an API: they're much harder to write to be reliable, and they can break at any time, whenever the site makes even the smallest change. APIs avoid all that mess, and make performance far better too (on both sides), since you're only downloading the data you want, not a ton of Javascript and HTML that you don't.
Re:Congrats! Web scraping is legal! (US precedent)
by barrkel on Wednesday January 29, 2020 @10:51AM
(#22181817)
APIs are often not as complete as the web interface, since the customer sees the web interface and normally the customer is what drives the revenue model of the company.
If pages are driven via an API, then the API is preferable, but publicly facing websites are often a mix of server-side HTML generation and API enrichment, for caching if nothing else.
Re:Congrats! Web scraping is legal! (US precedent)
by magduf on Wednesday January 29, 2020 @10:52AM
(#22181832)
In that case it seems that the webmasters complaining about scraping need to make sure their APIs actually provide access to all the same data, if they want people to use the APIs instead of scraping.
Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a consumer broadband IP.