News for hackers. Stuff that matters.

A comment on Ask HN: Website with 6^16 subpages and 80k+ daily bots. Parent comment

Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by 2000swebgeek on Thursday October 24, 2024 @05:50AM (#41933909)
better yet, see if bots access /robots.txt, find them from there. no human looks at robots.txt :)

add a captcha by limiting IP requests or return 429 to rate limit by IP. Using popular solutions like cloudflare could help reduce the load. Restrict by country. Alternatively, put in a login page which only solves the captcha and issues a session.

[ Parent | on HN ]
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by naikrovek on Thursday October 24, 2024 @09:11AM (#41935156)
> no human looks at robots.txt

of course people look at this. it's not an everyday thing for the prototypical web user, but some of us look at those a lot.

[ Parent | on HN ]
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by pkaeding on Thursday October 24, 2024 @11:46AM (#41936696)
Yes, but those of us also often have trouble solving captchas.
[ Parent | on HN ]
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by quectophoton on Thursday October 24, 2024 @09:12AM (#41935169)
> no human looks at robots.txt :)

I... I do... sometimes. Mostly curiosity when the thought randomly pops on my head. I mean, I know I might be flagged by the website as someone weird/unusual/suspicious, but sometimes I do it anyway.

Btw, do you know if there's any easter egg on Hacker News' own robots.txt? Because there might be.

[ Parent | on HN ]