There is tree-sitter work being done in a feature branch of the official Emacs git repo: https://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/t...
It's still very much a WIP, but the fact that it's there at all is promising.
I don't know what this fork brings to table, but you could try tree-sitter today with your vanilla Emacs using a package[1] that works via dynamic module.
Personally I am more interested in getting structural selection and navigation reliably working for any language. There is also a package named combobulate[2] to help with that.
Dunno about the overall ceiling of Emacs' syntax highlighting capabilities. But the few times in the past that I've had reason to look at the definition of the highlighting for specific languages, the Lisp code has looked like horrific imperative rats nests.
As with most things Emacs, there is no real ceiling to what you can do, because elisp. But as you observed, there's a reason that using syntax-table rules to write a highlighter has been called "black magic".
The sooner there is anything even remotely grammar-based for creating highlighters in Emacs, the better.
You can use tree-sitter already if you have dynamic module support: https://github.com/emacs-tree-sitter/elisp-tree-sitter
1) highlighting - most unnecessary thing people invented, except values and :things: in ""
2) LSP client provides these things exactly as other programs do.
GNU means do everything different because otherwise it is not GNU, so maybe thats not good thing after all.
This push is so strong even Steve Jobs started to Think Different after cloning BSD in 199x...
When I've mentioned this before someone usually explains why I am wrong and it is actually superior or whatever. I do not care I don't like it as much. It's nowhere near enough to drop emacs over but if this changes it I might prefer it.