News for hackers. Stuff that matters.
Re:How we scrape 300k prices per day from Google Flights by jsnell on Saturday June 13, 2020 @02:54PM (#23511714)
> This isn’t an astronomical number, but it’s large enough that we (at least, as a bootstrapped company) have to care about cost efficiency.

... by externalizing the costs to a third party.

In general, I'm really surprised that they published this article. It's like they described exactly the data that somebody working on preventing scraping would need to block this traffic, in totally unnecessary level of detail. (E.g. telling exactly which ASN this traffic would be arriving from, describing the very specific timing of their traffic spikes, the kind of multi-city searches that probably see almost no organic traffic).

I just don't get it. It's like they're intentionally trying to get blocked so that they can write a follow-up "how Google blocked our bootstrapped business" blog post.

Re:How we scrape 300k prices per day from Google Flights by fxtentacle on Saturday June 13, 2020 @03:02PM (#23511777)
Or they just don't understand that what they are doing is illegal.

I'm always surprised by the level of ignorance, but I've seen more than one startup burn because the founders didn't understand which taxes were due and, thus, failed to account for them in their pricing.

Re:How we scrape 300k prices per day from Google Flights by temp43t453 on Saturday June 13, 2020 @03:57PM (#23512203)
? https://news.ycombinator.com/item?id=22180559 i dont think it's unethical in the age of ad driven web
Re:How we scrape 300k prices per day from Google Flights by sdinsn on Saturday June 13, 2020 @09:11PM (#23514299)
Scraping public data is not illegal in the US.

> I'm always surprised by the level of ignorance

Such as the ignorance displayed in your comment?

Re:How we scrape 300k prices per day from Google Flights by meritt on Saturday June 13, 2020 @03:05PM (#23511808)
> what they are doing is illegal

It's not illegal. Google can sue them and bury them in court fees and potentially win a civil suit, but it sure as hell isn't illegal.

Re:How we scrape 300k prices per day from Google Flights by fxtentacle on Saturday June 13, 2020 @03:09PM (#23511842)
I'm pretty sure this is legal wrt CFAA because they are not circumventing any access control mechanism.

However, 300k requests per day surely is enough that this could be considered a sort of denial of service attack and violate fair use.

If Google wanted to, they would scale down their servers for a day, wait for this traffic peak to hit, document how it made the service unavailable to others, and now you have a valid offense to sue for.

Re:How we scrape 300k prices per day from Google Flights by tmerr on Saturday June 13, 2020 @04:03PM (#23512245)
I am surprised people think this could affect the servers? If they scrape 300k prices from 25k pages a day, and the crawl runs every hour from 15-22 UTC, then that means the 25k is spread across 3.5k pages per crawl. Even if the crawl is aggressive and completes within a minute that is still only 58 QPS.
Re:How we scrape 300k prices per day from Google Flights by occamrazor on Saturday June 13, 2020 @03:30PM (#23512005)
They are using residential proxies to evade rate-limiting by google. I don’t know if it’s enough to trigger CFAA, but it shows at a minimum that they know that what they are doing is not what google considers fair use.
Re:How we scrape 300k prices per day from Google Flights by bdcravens on Saturday June 13, 2020 @03:21PM (#23511928)
> what they are doing is illegal

illegal means violate criminal code.

> now you have a valid offense to sue for

That's civil, not criminal.

It's pedantic, but words matter.

Re:How we scrape 300k prices per day from Google Flights by fragmede on Saturday June 13, 2020 @04:23PM (#23512376)
The court decision in the hiQ vs LinkedIn case limits the CFAA's reach and scraping of publicly accessible data "likely does not violate the CFAA".

https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Re:How we scrape 300k prices per day from Google Flights by meritt on Saturday June 13, 2020 @03:14PM (#23511871)
> now you have a valid offense to sue for.

Yes. That's also what I said above when I said "Google can sue them". It's not illegal is my entire point. You can sue anyone, for any reason, at any time, and even win. That doesn't make something illegal.

If Google sends them a C&D, and expressly forbids them from doing this activity, and implements technical measures to prevent them from doing so, and they continue doing so then they may start approaching the area of illegal (Craigslist v. 3Taps would agree, hiQ v. LinkedIn would disagree).

Re:How we scrape 300k prices per day from Google Flights by sdinsn on Saturday June 13, 2020 @09:10PM (#23514289)
> denial of service attack

300k requests / day is a little over 3 per second. That's not much.

Re:How we scrape 300k prices per day from Google Flights by aardvark291 on Saturday June 13, 2020 @03:03PM (#23511794)
Unethical? Yes. Illegal? How?
Re:How we scrape 300k prices per day from Google Flights by sneak on Saturday June 13, 2020 @03:06PM (#23511825)
It’s not even unethical. Google publishes the data to the public for all to see.
Re:How we scrape 300k prices per day from Google Flights by digsy on Saturday June 13, 2020 @03:21PM (#23511930)
Unethical to build a business scraping data from a company that makes money scraping data?
Re:How we scrape 300k prices per day from Google Flights by aardvark291 on Saturday June 20, 2020 @06:10PM (#23587794)
The Google Flights data is not "scraped." They interface directly to the airline reservations systems.
Re:How we scrape 300k prices per day from Google Flights by splonk on Saturday June 13, 2020 @03:14PM (#23511878)
I agree with you in principle, but having worked on both sides of this, I think there's very little chance they get blocked at their current traffic levels.

I do think that if they ever get traction they'll have a lot of problems - there's a reason GDS access to flight availability is slow, expensive, and difficult to implement well. Scraping definitely won't scale.

Re:How we scrape 300k prices per day from Google Flights by sneak on Saturday June 13, 2020 @03:07PM (#23511829)
> E.g. telling exactly which ASN this traffic would be arriving from

The article mentions that they are using rotating residential proxies.

Re:How we scrape 300k prices per day from Google Flights by jsnell on Saturday June 13, 2020 @03:10PM (#23511853)
Ah, thanks! I missed that part in the writeup.
Re:How we scrape 300k prices per day from Google Flights by fsaintjacques on Saturday June 13, 2020 @03:23PM (#23511946)
There's a very high chance this is monetization of botnets, which adds even more to the overall shadyness.
Re:How we scrape 300k prices per day from Google Flights by dylz on Saturday June 13, 2020 @04:40PM (#23512513)
This is almost certainly guaranteed to be either monetization of botnets, or inadvertently installed adware.
Re:How we scrape 300k prices per day from Google Flights by fxtentacle on Saturday June 13, 2020 @03:10PM (#23511848)
Why would they need those proxies if what they are doing is fully legal?
Re:How we scrape 300k prices per day from Google Flights by asciident on Saturday June 13, 2020 @03:18PM (#23511913)
You do know that "legal" means "against the law" right? And when you say something is illegal, then you need to produce the law it violates; there's no proof from a company trying to prevent you from doing it.