News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by aopwiejfpoieajf on Thursday July 17, 2025 @11:34AM (#44594480)
This is how it's going. Half the websites I go to have Cloudflare captchas guarding them at this point. Every time I visit StackOverflow I get a 5 second wait while Cloudflare decides I'm kosher.
Re:I was wrong about robots.txt by Sohcahtoa82 on Thursday July 17, 2025 @02:58PM (#44596838)
Are you using TOR or a VPN, spoofing your User-Agent to something uncommon, or doing something else that tries to add extra privacy?

That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.

Re:I was wrong about robots.txt by EPendragon on Thursday July 17, 2025 @04:07PM (#44597599)
I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.
Re:I was wrong about robots.txt by ponooqjoqo on Thursday July 17, 2025 @11:16PM (#44600855)
I get this and I assume it's because I clear cookies pretty frequently. It used to be the case that that didn't matter, but nowadays everyone shields their websites using JS.
Re:I was wrong about robots.txt by 71bw on Friday July 18, 2025 @07:30AM (#44603507)
I use regular Edge with uBlock and get cloudflare crapchas all the time.
Re:I was wrong about robots.txt by tomrod on Sunday July 20, 2025 @12:33AM (#44622003)
I've found Vivaldi to be a much better experience, chrome based, and supports ublock in all its glory
Re:I was wrong about robots.txt by 71bw on Friday July 25, 2025 @06:03AM (#44681482)
I tried it and didn't like it. Currently migrating to Firefox though, their vertical tab implementation is really decent.