I was wrong about robots.txt

Re:I was wrong about robots.txt by Sohcahtoa82 on Thursday July 17, 2025 @02:58PM (#44596838)

Are you using TOR or a VPN, spoofing your User-Agent to something uncommon, or doing something else that tries to add extra privacy?

That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.

[ Parent | on HN ]

Re:I was wrong about robots.txt by EPendragon on Thursday July 17, 2025 @04:07PM (#44597599)

I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.

[ Parent | on HN ]

Re:I was wrong about robots.txt by ponooqjoqo on Thursday July 17, 2025 @11:16PM (#44600855)

I get this and I assume it's because I clear cookies pretty frequently. It used to be the case that that didn't matter, but nowadays everyone shields their websites using JS.

[ Parent | on HN ]

Re:I was wrong about robots.txt by 71bw on Friday July 18, 2025 @07:30AM (#44603507)

I use regular Edge with uBlock and get cloudflare crapchas all the time.

[ Parent | on HN ]

Re:I was wrong about robots.txt by tomrod on Sunday July 20, 2025 @12:33AM (#44622003)

I've found Vivaldi to be a much better experience, chrome based, and supports ublock in all its glory

[ Parent | on HN ]

Re:I was wrong about robots.txt by 71bw on Friday July 25, 2025 @06:03AM (#44681482)

I tried it and didn't like it. Currently migrating to Firefox though, their vertical tab implementation is really decent.

[ Parent | on HN ]