That said, as an experiment, I did about half of my Master's thesis work in TeX. Without quick feedback on the rendered formulae, I had a much harder time reasoning through things, so having a comfortable text editing environment seems crucial.
That said, I wouldn't recommend using TeX as a scratchpad. It just doesn't map well to the sets of tools that we tend to use: mini diagrams, arrows between parts of the page, strikeout and circling text, etc.
What might work is something like e-paper that renders a handwritten document into text. Ideally, we could turn letters into Unicode and diagrams into vector graphics, providing tools to edit and embellish the rendered versions if desired.
It's extremely common to edit equations and diagrams on the fly, so I would probably feel friction with anything other that doesn't seem to just magically "prettify" whatever I handwrite at it. Seems like a really tall order though.
The standard notation is mostly for communicating with other people mediated via papers. But we don't actually use that many brackets in math---that's more of a programming thing. (Because programmers can't really use visual structure all that much, and mostly rely on linearization of their syntax trees. Languages with significant whitespace like Haskell (and to a lesser extent Python) see less parens and braces.)
[0] https://en.m.wikipedia.org/wiki/Penrose_graphical_notation
https://www.youtube.com/watch?v=sULa9Lc4pck
A bit lowbrow, perhaps (you'd want Penrose notation or such things in that case).
(Uncreative, probably very bad) examples: Replace brackets with boxes, use colors (think syntax highlighting), icons, or even basic text-like formatting. And that's still being very conservative in relation to what could be done, as exemplified by the OP.
Any pointers/ideas?