Re:Ask HN: What do people use to prevent crawlers?
by peterkelly on Sunday July 23, 2017 @07:05AM
(#14831723)
https://en.wikipedia.org/wiki/Robots_exclusion_standard
[ Parent
| on HN ]
Re:Ask HN: What do people use to prevent crawlers?
by AmrMostafa on Sunday July 23, 2017 @08:22AM
(#14831932)
robots.txt is a "sign" that you leave at the door for well-behaving robots, but it doesn't actually make any practical difference when a robot isn't implemented to honor it