That said, as an experiment, I did about half of my Master's thesis work in TeX. Without quick feedback on the rendered formulae, I had a much harder time reasoning through things, so having a comfortable text editing environment seems crucial.

That said, I wouldn't recommend using TeX as a scratchpad. It just doesn't map well to the sets of tools that we tend to use: mini diagrams, arrows between parts of the page, strikeout and circling text, etc.

What might work is something like e-paper that renders a handwritten document into text. Ideally, we could turn letters into Unicode and diagrams into vector graphics, providing tools to edit and embellish the rendered versions if desired.

It's extremely common to edit equations and diagrams on the fly, so I would probably feel friction with anything other that doesn't seem to just magically "prettify" whatever I handwrite at it. Seems like a really tall order though.