Re:I was wrong about robots.txt
by aopwiejfpoieajf on Thursday July 17, 2025 @11:34AM
(#44594480)
This is how it's going. Half the websites I go to have Cloudflare captchas guarding them at this point. Every time I visit StackOverflow I get a 5 second wait while Cloudflare decides I'm kosher.
[ Parent
| on HN ]
Re:I was wrong about robots.txt
by Sohcahtoa82 on Thursday July 17, 2025 @02:58PM
(#44596838)
Are you using TOR or a VPN, spoofing your User-Agent to something uncommon, or doing something else that tries to add extra privacy?
That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.
Re:I was wrong about robots.txt
by EPendragon on Thursday July 17, 2025 @04:07PM
(#44597599)
I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.
Re:I was wrong about robots.txt
by ponooqjoqo on Thursday July 17, 2025 @11:16PM
(#44600855)
I get this and I assume it's because I clear cookies pretty frequently. It used to be the case that that didn't matter, but nowadays everyone shields their websites using JS.
I use regular Edge with uBlock and get cloudflare crapchas all the time.
I've found Vivaldi to be a much better experience, chrome based, and supports ublock in all its glory