I think this would be pretty easy to do in eshell. Emacs gives you frames already and supports interacting with output buffers (like compilations errors)

Its been on my to-do list for a while. I wrote a rough outline and its similar to what you've said as well: https://reddit.com/comments/6y3q4k/comment/dml16vq