Re:Robots.txt for the NYT has a specific exclusion for an 19...
by michaelcampbell on Monday March 29, 2021 @07:20AM
(#26620047)
I have a specific exclusion in my robots.txt file, and also a cron-scheduled grep of my logs to see if anything actually hits it. I don't care if they do or they don't, but it's a way for me to exclude specific bots that don't honor my robots.txt file.
[ Parent
| on HN ]
There are no comments on this comment yet.