Usually starts from simple to difficult. User agent stuff, IP address detection, aggressive rate limiting, captcha checking, browser fingerprinting, etc.

It can be overcome and, admittedly, I am new to this so for me that means way more time spent trying to make it work. The odd one that I got stuck on for a while was a presented list where individual record held pertinent details, but the list had, seemingly randomly, items that looked like records on the surface, but were not ( so those had to be identified and ignored ). Small things like that.

Still, I would love to learn more about your approach if you would be willing to share.

Once recaptcha is in the mix it'll get tricky pretty quickly. Everything else is easy to overcome most of the time.

Recaptcha comes to mind.

That said, there are quite a few services which battle these systems for you nowadays (such as scraperapi - not affiliated, not a user). They are not always successful, but they have an advantage of maaany residential proxies (no doubt totally ethically obtained /s, but that's another story).

please click on all the traffic lights you see below.

Datadome, Incapsula