Re:Fewer Than Half of Google Searches Now Result in a Click
by SahAssar on Wednesday August 14, 2019 @07:09PM
(#20700945)
They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking?
Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.
they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.