Re:Tusk – Short-lived anonymous feedback
by mrkramer on Monday February 08, 2021 @05:28AM
(#26062815)
A lot of webmasters are amateurs who expose their credentials even when robot rules are respected. Google dorking is a way to find that credentials and other sensitive data and information.
Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.
Their docs say they do:
https://developers.google.com/search/reference/robots_txt
They do say that they ignore robots.txt for user-initiated actions eg. Google Translate, which makes sense in my opinion.
Products like Google Search and Bing have a self-interest in supporting robots.txt because the alternative is annoyed webmasters suing them for not providing an opt-out due to which their unsecured credentials are a quick search away.
In fact, I believe the robots.txt protocol was devised in consultation with search engines.