News for hackers. Stuff that matters.
Re:Avoiding bot detection: How to scrape the web without get... by dzhiurgis on Monday November 01, 2021 @02:17AM (#29063537)
Do you know much about airline api pricing more?

I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.

Re:Avoiding bot detection: How to scrape the web without get... by oxymoron on Monday November 01, 2021 @03:00AM (#29063774)
Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight search software vendors have dedicated teams for this.
Re:Avoiding bot detection: How to scrape the web without get... by namdnay on Monday November 01, 2021 @03:53AM (#29064043)
In fact the airlines are charged per book, but if and only if the look to book stays within reasonable bounds. If it rockets up, they’re on the hook for the penalties
Re:Avoiding bot detection: How to scrape the web without get... by oxymoron on Monday November 01, 2021 @04:08AM (#29064130)
Thanks for the clarification! I probably misremember some of the details.