Google acquired Metaweb Technologies in 2010, acquiring Freebase with it. Freebase was a semantic web knowledge base and this became deeply integrated into Google's search technology. They did, in fact, want to push semantic web attributes to make the web more indexable, even though they originated neither the bigger idea nor the original implementation.
(one of my classmates ended up as an engineer at Metaweb, then Google)
This sounds more like "indexing" than "crawling"
The "Sitemaps 0.84" protocol , e.g., sitemap.xml, was another standard that _was_ introduced by Google
Helpful for crawling and other things
(I convert sitemap.xml to rss; I also use it to download multiple pages in a single TCP connection)
Not every site includes a sitemap, some do not even publish a robots.txt
Kudos. I respect this kind of honesty. I wish there was more of it.
W3C of course deserves credit for their hard work on this standard.
My main point was that regardless of the semantic "standard", nothing prevented us from putting everything in a generic div, so complaining that everyone's just "not on board" isn't a useful lament.