News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by NoMoreNicksLeft on Thursday July 17, 2025 @09:38AM (#44593273)
I now write all of my bots in javascript and run them from the Chrome console with CORS turned off. It seems to defeat even Google's anti-bot stuff. Of course, I need to restart Chrome every few hours because of memory leaks, but it wasn't a fun 3 days the last time I got banned from their ecosystem with my kids asking why they couldn't watch Youtube.
Re:I was wrong about robots.txt by tomrod on Sunday July 20, 2025 @12:39AM (#44622029)
Where can I learn more about custom bots in JS and Chrome?