Europeans used to write in Latin, which is now incomprehensible to most, and yet, it has all worked out.

But granted, the transition took a while and was not just a matter of habit.

Having said that, why could one not publish important classic texts in a phonetic vernacular version, and leave the study of the originals to those with the inclination (just as most people in the West do not learn Hebrew, ancient Greek, and Latin; yet can find versions of the Bible, Euclid, Seneca, and Newton that they can read if they so desire.)

You're drawing a false analogy here, the problem isn't that phonetic writing isn't capable of rendering classical texts; the problem is that they can't competently render any text written in the formal register.

So, here we get to the fundamental issue then: Chinese seems to have a larger divergence between spoken and written register than most other languages (particularly alphabetically written ones).

It's unclear to me how far that extends. (This issue seems intertwined with the issue that many speakers of other Chinese "dialects"/"languages" also use Standard Written Chinese (ie Mandarin), btw.)

It is possible to write some register of Chinese in phonetic writing, though - that's shown by Dungan (written in Cyrillic), by the experiments in the 30s (quoting Mao again: "we have begun experimenting with Hsin Wen Tzu—Latinized Chinese. It is now used in our Party school, in the Red Academy, in the Red Army, and in a special section of the Red China Daily News."), etc.

Characters can be mapped to their phonetic equivalents no?

Yeah, but not back (in other words, a mechanical mapping won't preserve sufficient information to disambiguate). It's definitely not trivial.

But they can't be mapped back.