Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Monday November 01, 2021 @04:56AM (#29064356)

Usually means they have large ongoing problems with aggressive bots.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Monday November 01, 2021 @05:18AM (#29064494)

Or they using it as lazy approach to fix long-dos attacks by competitors.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Monday November 01, 2021 @07:41AM (#29065439)

What is the non-lazy alternative that stops a botnet DDOS without affecting regular visitors?

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Tuesday November 02, 2021 @06:12AM (#29079135)

Long-dos is usually not a botnet but a single or a few attackers requesting a resource heavy page on and on or using a technic like slowloris. If you are using WordPress I am sure you find some signs in your log.

The solution other than with ddos is to make the app better. Caching, access control and limiting, etc.

[ Parent | on HN ]