Re:Common Robots. Txt Pitfalls and How to Avoid Them
by AnnYaroshenko on Monday June 24, 2019 @04:44AM
(#20261578)
1. Ignoring disallow directives for specific user-agent block;
2. One robots.txt file for different subdomains;
3. Listing of secure directories;
4. Blocking relevant pages;
5. Forgetting to add directives for specific bots where itβs needed;
6. Adding the relative path to sitemap;
7. Ignoring Slash in a Disallow field;
8. Forgetting about case sensitivity.
[ Parent
| on HN ]
There are no comments on this comment yet.