I think it should be an application programming language and only source code and assets should be transmitted. Crawlable content should be provided by a function in the language, if the author of the document/app decides so.

A good browser/server/IDE would both be easier and more powerful than any of the messy mixes of languages and document formats we use right now.