Rate limiting isn't an effective defense for us.

First, as a B2B site, many of our users from a given customer (and with huge customers, that can be many) are coming through the same proxy server, effectively presenting to us as the same IP,

Second, the bots years back became much more sophisticated than a single, or even relatively finite, IP. Today they work AWS, Azure, GCP, and other cloud services. So the IPs that they're assigned today will be different tomorrow. Worse, the IPs that they're assigned today may well be used by a real customer tomorrow.

If your users are logged in you can rate limit by user instead of by IP. This mostly solves this problem. Generally what I do is for logged in users I rate limit by user, then for not-logged-in users I rate limit aggressively by IP. If they hit the limit the message lets them know that they can get around it by logging in. Of course this depends on user accounts having some sort of cost to create. I've never actually implemented it but considered having only users who have made at least one purchase bypass the IP limit or otherwise get a bigger rate limit.

Have you tried including the recaptcha v3 library and looking at the distribution of scores? -- https://developers.google.com/recaptcha/docs/v3 -- "reCAPTCHA v3 returns a score for each request without user friction"

It obviously depends on how motivated the scrapers are (i.e. whether their headless browsers are actually headless, and/or doing everything they can to not appear headless, whether Google has caught on to their latest tricks etc. etc.) but it would at least be interesting to look at the score distribution and then see whether you can cut off or slow down the < 0.3 scoring requests (or redirect them to your API docs)

For web scraping specifically, I’ve developed key parts of commercial systems to automatically bypass reCAPTCHA, Arkose Labs (Fun Captcha), etc.

If someone dedicated themselves to it, there’s a lot more that these solutions could be doing to distinguish between humans and bots, but it requires true specialized talent and larger expenses.

Also, for a handful of the companies which make the most popular captcha solutions, I don’t think the incentives align properly to fully segregate human and bot traffic at this time.

I think we’re still very much still picking at the very lowest hanging fruit, both for anti-bot countermeasures and anti-anti-bot (counter-countermeasures).

Personally I believe this will finally accelerate once AI’s can play computer games via a camera, keyboard, and mouse. And when successors GPT-3 / PaLM can participate well in niche discussion forums like HackerNews or the Discord server for Rust.

Until then it’s mainly a cost filter or confidence modification. As long as enough bots are blocked so that the ones which remain are technically competent enough to not stress the servers, most companies don’t care. And as long as the businesses deploying reCAPTCHA are reasonably confident that most of the views they get are humans (even if that belief is false), Google doesn’t have a strong incentive to improve the system.

Reddit doesn’t seem to care much either. As long as the bots which participate are “good enough”, it drives engagement metrics and increases revenue.

Scrapers can pay a commercial service to Mechanical Turk their way through reCAPTCHA. It makes a meaningful difference to scraping costs at scale, but sometimes it's still profitable.

I'd pay for a service to do this for me as an ordinary end user, so i never have to solve a captcha myself again.

You would still have to wait for each captcha to be solved, which might be more frustrating than doing it yourself.

It sounds great, until you have Chinese customers. That’s when you’ll figure out Recaptcha just doesn’t really work in China, and have to begrudgingly ditch it altogether…

Do you know if there's a way to rate limit logged-in users differently than visitors of a site?

rate limiting can be a double edged sword, you can be better off giving a scraper highest bandwidth so they are gone sooner, otherwise somthing like making a zip or other sort of compilation of the site available may be an option.

just what kind of scraper you have is a concern.

does scraper just want a bunch of stock images;

or does scraper have FOMO on web trinkets;

or does scraper want to mirror/impersonate your site.

the last option is the most concerning because then;

scraper is mirroring bcz your site is cool and local UI/UX is wanted;

or is scraper phishing smishing or otherwise duping your users.

Yeah, good points to consider. I think the sites that would be scrapped the most would be where the data is regularly and reliably up-to-date, and a large volume of it at that - so not just one scraper but many different parties may on a daily or weekly basis try to scrap every page.

I feel that ruling should have the caveat that if a fair cost paid API version for getting publicly listed data then the scrapers must legally use that (say no more than 5% more than cost of CPU/bandwidth/etc of the scraping behaviour); ideally a rule too that at minimum there be a delay if they are republishing that data without your permission, so at least you as the platform/source/reason for the data being up-to-date aren't harmed too - which may then kill the source platform over time if regular visitors somehow start going to the competitor publishing the data.

Absolutely you just have to check the session cookie

nginx can be set up to do that using the session cookie.