News for hackers. Stuff that matters.
Re:Ask HN: What do people use to prevent crawlers? by peterkelly on Sunday July 23, 2017 @07:05AM (#14831723)
Re:Ask HN: What do people use to prevent crawlers? by AmrMostafa on Sunday July 23, 2017 @08:22AM (#14831932)
robots.txt is a "sign" that you leave at the door for well-behaving robots, but it doesn't actually make any practical difference when a robot isn't implemented to honor it