Re:Ask HN: What do people use to prevent crawlers?
by AmrMostafa on Sunday July 23, 2017 @08:22AM
(#14831932)
robots.txt is a "sign" that you leave at the door for well-behaving robots, but it doesn't actually make any practical difference when a robot isn't implemented to honor it
[ Parent
| on HN ]
There are no comments on this comment yet.