Re:How we scrape 300k prices per day from Google Flights
by tmerr on Saturday June 13, 2020 @04:03PM
(#23512245)
I am surprised people think this could affect the servers? If they scrape 300k prices from 25k pages a day, and the crawl runs every hour from 15-22 UTC, then that means the 25k is spread across 3.5k pages per crawl. Even if the crawl is aggressive and completes within a minute that is still only 58 QPS.
Re:How we scrape 300k prices per day from Google Flights
by occamrazor on Saturday June 13, 2020 @03:30PM
(#23512005)
They are using residential proxies to evade rate-limiting by google. I don’t know if it’s enough to trigger CFAA, but it shows at a minimum that they know that what they are doing is not what google considers fair use.
Re:How we scrape 300k prices per day from Google Flights
by bdcravens on Saturday June 13, 2020 @03:21PM
(#23511928)
> what they are doing is illegal
illegal means violate criminal code.
> now you have a valid offense to sue for
That's civil, not criminal.
It's pedantic, but words matter.
Re:How we scrape 300k prices per day from Google Flights
by fragmede on Saturday June 13, 2020 @04:23PM
(#23512376)
The court decision in the hiQ vs LinkedIn case limits the CFAA's reach and scraping of publicly accessible data "likely does not violate the CFAA".
https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
Re:How we scrape 300k prices per day from Google Flights
by meritt on Saturday June 13, 2020 @03:14PM
(#23511871)
> now you have a valid offense to sue for.
Yes. That's also what I said above when I said "Google can sue them". It's not illegal is my entire point. You can sue anyone, for any reason, at any time, and even win. That doesn't make something illegal.
If Google sends them a C&D, and expressly forbids them from doing this activity, and implements technical measures to prevent them from doing so, and they continue doing so then they may start approaching the area of illegal (Craigslist v. 3Taps would agree, hiQ v. LinkedIn would disagree).
However, 300k requests per day surely is enough that this could be considered a sort of denial of service attack and violate fair use.
If Google wanted to, they would scale down their servers for a day, wait for this traffic peak to hit, document how it made the service unavailable to others, and now you have a valid offense to sue for.