"The authors even had to enrich XML with additional entities named “annotations” to make it suitable for their needs"
This is what Etherpad did for adding support for lists, bold, italics, etc. Did anyone complain? Does it not work?
Granted, Wave's approach is way more ambitious. They support any sort of XML groves, and not just simple 1-level deep tags like Etherpad.
I do not attack annotations or XML per se. The general approach "neither thing is precisely OK, so let's use both" seems questionable to me. I believe, XML is complex enough, so why do we need XXML? Finally, by using annotations they recognize that XML nesting is a pain in the ass, but they did not solve the problem, just sort of relieved it.
Actually, I doubt a lot, whether XML is actually carrying some useful load in the Wave. But the last one is just a feeling.
It seems to solve the issues that concern you (validating the correctness of OT).
What I will concede is that their OT implementation is probably not perfect yet. If you read the comments on some transformation functions on their Java code, they shown concern and do extra logging in hopes to track unexpected issues they haven't solved yet.
OT is a dead end. The problems they struggle with could be trivially bruteforced by employing unique symbol identifiers. http://bouillon.math.usu.ru/articles/ctre.pdf