Re:Creepy Crawlies
by robotmay on Monday August 31, 2026 @03:15AM
(#49506602)
It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.
[ Parent
| on HN ]
There are no comments on this comment yet.