Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by pzmarzly on Thursday October 24, 2024 @04:19AM
(#41933310)
There are some browser plugins that try to guess what technologies are used by the website you are visiting. I hope the better ones can guess it by just looking at HTML and HTTP headers, but wouldn't be surprised if others were querying some known endpoints.
[ Parent
| on HN ]
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by etiennebausson on Thursday October 24, 2024 @05:10AM
(#41933641)
Then they are, by definition, bots scriping the site for informations, and should start by the robots.txt