News for hackers. Stuff that matters.

A comment on How we scrape 300k prices per day from Google Flights.

Re:How we scrape 300k prices per day from Google Flights by randombytes6869 on Saturday June 13, 2020 @03:18PM (#23511909)
To those lamenting that they're scraping... Google is the biggest scraper of them all. Facebook, Amazon, Google, Microsoft. All the big boys scrape voraciously, yet try their best to block themselves from being scraped. Scraping is vital for the functionality of the internet. The narrative that scraping is evil is what big companies want you to think.

When you block small scrapers from your site but permit giants like Googlebot and Bing all you're doing is locking in a monopoly that's bad for everyone

[ Parent | on HN ]
Re:How we scrape 300k prices per day from Google Flights by occamrazor on Saturday June 13, 2020 @03:33PM (#23512034)
Google has the (often implicit) permission of the website owner to scrape. OTOH, Google Flights explicitly disallows scraping results.
[ Parent | on HN ]
Re:How we scrape 300k prices per day from Google Flights by sdinsn on Saturday June 13, 2020 @09:17PM (#23514326)
No, Google's scraping is opt-out only, which they offer to be friendly.

Google does not need anyone's permission to scrape publicly accessible data, and they are not required to follow any opt-out requests.

[ Parent | on HN ]