That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.
Recaptcha has been almost useless, in my experience. If you read the spam logs, you'll quickly learn about the spam software they (claim to) use to bypass Recaptcha, because that's what they end up promoting. I started tagging in logs if Recaptcha had validated on messges, and sure enough these spam posts had all successfully passed it. Great opportunity to rip out more Google dependencies from my website.
I've found my own custom written filters to be vastly more effective than Recaptcha.
Lots of the bots are running full Chrome with JS, lots of HeadlessChrome being used lately. The fact that they're using HeadlessChrome is something that makes them easy to detect, ahem.
Recaptcha was also filtering out some legit humans (I logged all posts regardless of captcha status to be reviewed later), so it just wasn't worth reducing the user experience when the captcha bot detection rate was so low.
I would add to that:
- block signups/comments from known throwaway email domains
- block known datacenter IP ranges, at least for POST requests. Honestly on our sites 50% of spam was coming from AWS EC2 IPs
- use a proxy/vpn/bot detection service like https://focsec.com
The 0.001% was that person using EC2 as a proxy or VPN server.
Guess the fraudsters were vendor locked lol.
Throwaway email services have caught up to the point where even bigger players don't manage to block them, i.e. they rotate the domains used for the emails.
because then the attack becomes DOS as they cycle through dictionaries of words?
First, of course, you have cloudflare and recaptcha, which are free and very efficient, as the author say.
But even if you don't want to use them (some of my services don't), most bots are very dumb:
- require JS, and you lose half of the web ones
- silly tricks like hidden input fields in forms that worked in 2000 still work in 2022. Use a bunch of them, and you can yet again halve the bot traffic.
- many URL should have impossible to guess paths. E.G: just changing the /admin/ url to a uuid in django or the /wp-admin/ in wordpress, you save so many requests.
- bots are usually not tailored to your site, meaning if you require JS, you can actually embed anti-bot measure in the client code and they will work. E.G: exponential backoff + some heavy calculations if too many fast consecutive ajax requests.
- fail2ban + a few iptables rules (mitigate syn flood, etc) will help
- varnish + redis gets you very far to shave excess dummy traffic
It's not great, but it's not an apocalypse.
Unless you are under targeted attack.
Then it sucks and you die.