Re:A flight search engine that combines flights from differe...
by residentraspber on Friday September 20, 2024 @02:47PM
(#41604543)
Some airlines (Air Canada, in particular) are ramping up their attacks outside of just technical means [1].
It's a pretty sad state of affairs, but it seems some airlines would rather throw lawyers at a problem than let their users have a better experience.
[1]: https://onemileatatime.com/news/airlines-shut-down-websites-...
Re:A flight search engine that combines flights from differe...
by tomhoward on Friday September 20, 2024 @07:37PM
(#41606393)
We built the company from the start by scraping low cost airlines, and always did a lot of it.
It just a very big undertaking - building all the scrapers, maintaining them (every time any detail of an airline website changed), getting around anti-scraping measures, running servers, and even then your data is out of date as soon as a flight’s price or availability changes.
Out of curiosity, what's stopping you from just scraping/crawling this on a daily basis? I'm at a scale-up that offers a BI solution for the hospitality sector, and we scrape well into the tens of millions of data points per day using web crawlers.
They obviously don't want you to, so it's an ongoing cat and mouse game of integrations breaking, but once we we're established and had a customer base, the ability to negotiate data API agreements actually opened up. Especially if you have data to offer in return.