I'm not sure if we're getting hung up on the word "complex", but here's 60+ pages on paragraph line breaking alone:
http://www.eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf
Digital Typography is 700 pages long (not everything is going to be TeX fundamentals, but...):
https://archive.org/details/digitaltypograph0000knut/page/n7...
The book "Digital Typography" is, somewhat misleadingly, not a book that was written about digital typography, but simply a collection (https://cs.stanford.edu/~knuth/selected.html) of several papers that Knuth wrote about TeX and Metafont and some related topics, over several decades.
Maybe the several hundred pages of the "Computers and Typesetting" series would be a better example for your point: https://cs.stanford.edu/~knuth/abcde.html
In any case, I agree with the comment you're replying to: if you use TeX itself without dragging behind you the entire ecosystem and all the packages (in particular, use plain TeX rather than LaTeX) you'll see it's not really complex, and the implementation is fine. Especially today, with things like LuaTeX scripting (and, say, opTeX) it is feasible to bypass LaTeX and packages and do things yourself — that is, unless external circumstances require you to use LaTeX and packages etc, which is still often the case (e.g. submitting papers to journals).
Feel free to come up with something new that supports Markdown as a subset or overlaps with Markdown.
Markdown as-is is lovely.
I think the best future option is going to be djot[0]. It is being created by the author of pandoc, who might possibly be the most qualified person in the world to appreciate all if the nuances of marking up text and parsing it.
The rationale page can do a better job than myself. https://github.com/jgm/djot#rationale
: echo "word**withboldtext**inside" | pandoc
<p>word<strong>withboldtext</strong>inside</p>
: echo "a*?*b" | pandoc
<p>a<em>?</em>b</p>(which is why markdown still doesn't officially have any maths support, only certain non-standard flavours do)
Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.