Re:I was wrong about robots.txt
by EPendragon on Thursday July 17, 2025 @09:38AM
(#44593264)
For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable for the rest of the current "free billing" cycle, i.e. until your quota gets renewed.
[ Parent
| on HN ]
There are no comments on this comment yet.