Re:Ask HN: What should we do if someone is crawling our webs...
by elmerfud on Monday May 18, 2020 @03:37PM
(#23227353)
Is the crawling not respecting your robots file? You don't state a reason for wanting to block the behavior other than it was fun to build the tooling to discover it for you.
Personally I don't see a reason to block non abusive things from crawling a site. Sites are there to be found and read. Indexers, archivers, etc... are normal things that may provide a non obvious benefit.
This seems to be a judgement call on if you believe the the actions are nefarious or not.
Personally I don't see a reason to block non abusive things from crawling a site. Sites are there to be found and read. Indexers, archivers, etc... are normal things that may provide a non obvious benefit.
This seems to be a judgement call on if you believe the the actions are nefarious or not.