News for hackers. Stuff that matters.
Re:Ask HN: What should we do if someone is crawling our webs... by elmerfud on Monday May 18, 2020 @03:37PM (#23227353)
Is the crawling not respecting your robots file? You don't state a reason for wanting to block the behavior other than it was fun to build the tooling to discover it for you.

Personally I don't see a reason to block non abusive things from crawling a site. Sites are there to be found and read. Indexers, archivers, etc... are normal things that may provide a non obvious benefit.

This seems to be a judgement call on if you believe the the actions are nefarious or not.

There are no comments on this comment yet.