Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.
I'm not sure if we're getting hung up on the word "complex", but here's 60+ pages on paragraph line breaking alone:
http://www.eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf
Digital Typography is 700 pages long (not everything is going to be TeX fundamentals, but...):
https://archive.org/details/digitaltypograph0000knut/page/n7...
The book "Digital Typography" is, somewhat misleadingly, not a book that was written about digital typography, but simply a collection (https://cs.stanford.edu/~knuth/selected.html) of several papers that Knuth wrote about TeX and Metafont and some related topics, over several decades.
Maybe the several hundred pages of the "Computers and Typesetting" series would be a better example for your point: https://cs.stanford.edu/~knuth/abcde.html
In any case, I agree with the comment you're replying to: if you use TeX itself without dragging behind you the entire ecosystem and all the packages (in particular, use plain TeX rather than LaTeX) you'll see it's not really complex, and the implementation is fine. Especially today, with things like LuaTeX scripting (and, say, opTeX) it is feasible to bypass LaTeX and packages and do things yourself — that is, unless external circumstances require you to use LaTeX and packages etc, which is still often the case (e.g. submitting papers to journals).
Feel free to come up with something new that supports Markdown as a subset or overlaps with Markdown.
Markdown as-is is lovely.
I think the best future option is going to be djot[0]. It is being created by the author of pandoc, who might possibly be the most qualified person in the world to appreciate all if the nuances of marking up text and parsing it.
The rationale page can do a better job than myself. https://github.com/jgm/djot#rationale
: echo "word**withboldtext**inside" | pandoc
<p>word<strong>withboldtext</strong>inside</p>
: echo "a*?*b" | pandoc
<p>a<em>?</em>b</p>(which is why markdown still doesn't officially have any maths support, only certain non-standard flavours do)
As far as I could get, Sile is exactly that.
I'd prefer one of these:
- A human-writable text format that's displayed in a formatted way, eg TeX or MathJax
- A binary format that represents structs and enums in code with a clear documentation of how it's packed and unpacked. That can be formatted for reading, and perhaps has a TeX-like API (or is just constructed programmatically using an API)
The fundamentals of TeX's typesetting are amazing, but everything else is just bolted on. You'd probably want to rebuild it with its own, custom language, perhaps inspired more by modern XML/markdown rather than the old TeX language.
The biggest problem that we're going to have moving past TeX is that it's something of a standard for math representation in text. MathML never _really_ took off. Re-training millions of people skilled in typesetting math is going to be _tough_.