Strong disagree. Since robots.txt is optional and the default is "crawl me as you please", website owners don't "choose to make it available", they just don't choose to make it non-available.
[1] https://www.practicalecommerce.com/Search-Engines-Indexing-a...
[2] https://www.alrc.gov.au/publication/copyright-and-the-digita...
Also google cache allows users to get a copy of your site without visiting your site.
Why can they republish your data while we cannot? Why do we have to opt-out?
Maybe also get Google to update their docs: https://developers.google.com/search/docs/crawling-indexing/...
https://developers.google.com/search/docs/crawling-indexing/...
If you have specified in your robots.txt that you do not want the page(s) or directories ingested then only the url is indexed (if it is linked from another page). It does prevent the public display of the content of a page and creation description/summary.
“While Google won't crawl or index the content blocked by a robots.txt file”
They will show the URL if someone else has linked to it. But the content itself is not indexed.
I remember using Google the day it went public and it had no ads which made it unique compared to Altavista.
Wrong.
a) Robots.txt which defines what content you wish to make available to third parties predates every search engine including Google. Web site owners chose to make it available to Google and search engines have respected their wishes despite it not being in their best interest.
b) The difference here is that OpenAI, Meta etc have not even tried to honour the wishes of copyright holders. They just considered everything as theirs.
c) Google grew big because it had no ads, fast interface and PageRank was significantly better. It wasn't because it had the most comprehensive index.