Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by oefrha on Sunday October 31, 2021 @11:51PM (#29062830)

> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.

Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Monday November 01, 2021 @04:56AM (#29064356)

Usually means they have large ongoing problems with aggressive bots.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Monday November 01, 2021 @05:18AM (#29064494)

Or they using it as lazy approach to fix long-dos attacks by competitors.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Monday November 01, 2021 @07:41AM (#29065439)

What is the non-lazy alternative that stops a botnet DDOS without affecting regular visitors?

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Tuesday November 02, 2021 @06:12AM (#29079135)

Long-dos is usually not a botnet but a single or a few attackers requesting a resource heavy page on and on or using a technic like slowloris. If you are using WordPress I am sure you find some signs in your log.

The solution other than with ddos is to make the app better. Caching, access control and limiting, etc.

[ Parent | on HN ]