News for hackers. Stuff that matters.
Re:Creepy Crawlies by robotmay on Monday August 31, 2026 @03:15AM (#49506602)
It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.

There are no comments on this comment yet.