News for hackers. Stuff that matters.
Re:Common Robots. Txt Pitfalls and How to Avoid Them by AnnYaroshenko on Monday June 24, 2019 @04:44AM (#20261578)
1. Ignoring disallow directives for specific user-agent block; 2. One robots.txt file for different subdomains; 3. Listing of secure directories; 4. Blocking relevant pages; 5. Forgetting to add directives for specific bots where it’s needed; 6. Adding the relative path to sitemap; 7. Ignoring Slash in a Disallow field; 8. Forgetting about case sensitivity.

There are no comments on this comment yet.