News for hackers. Stuff that matters.

A comment on Ask HN: Website with 6^16 subpages and 80k+ daily bots. Parent comment

Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by etiennebausson on Thursday October 24, 2024 @05:10AM (#41933641)
Then they are, by definition, bots scriping the site for informations, and should start by the robots.txt
[ Parent | on HN ]

There are no comments on this comment yet.