Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by aembleton on Monday November 01, 2021 @05:10AM (#29064429)

I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices.

For example:

  curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false&FlexDaysBeforeOut=2&FlexDaysOut=2&ToUs=AGREED" | jq

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by jonatron on Monday November 01, 2021 @05:29AM (#29064563)

I haven't tried to scrape Ryanair, you could be right that it's trivial. It's the legal side that has a long and interesting history. Personally I wouldn't scrape them unless it was when working for a company that can afford lawyers.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by shapefrog on Monday November 01, 2021 @05:34AM (#29064590)

{"code":"TermsOfUseAreNotAccepted","message":"TermsOfUseAreNotAccepted"}

[ Parent | on HN ]