Meanwhile, in ssh, if you're behind a very bad connection and you cat a large file, your TCP bandwidth gets throttled down, but it still has to send the entire file over the connection, driving performance way down.
There is a fork of mosh (https://github.com/4ast/mosh/commits/master , primarily the "use UserStream instead of Terminal" commit) which disables the frame-sync behavior and just uses the byte-stream-sync code (which is used for keyboard data) to sync the raw bytes / ANSI escape codes back to the client. I haven't tried this, but my guess is that you'd be very sad once you accidentally catted a large file, but if you manage to never do that, you'll get mosh-like behavior but also scrollback.
I think you could also change the mosh model to involve maintaining scrollback server-side and syncing it (so you'd render and sync a (80+10000)x25 array, not just the last 80 lines), possibly at lower priority than the non-scrollback portion of the screen. I don't know if anyone's tried implementing this.
(Personally I just use screen anyway, so this isn't a major pain point for me)
Mosh syncing that 150x80 screen is a superb idea. Udp is great for mobile hotspot connections. However mosh server should save a buffer of 150x10000 on server side. Only sync 150x80 but If user uses native scroll then it should ask server for last X lines depending on scroll position.
If you want non-native scroll, just nest mosh inside screen/tmux.
(That may actually be why my theory doesn't work: once a line has scrolled off the screen, there's no way to update it, I think. So even if mosh wanted to do background, low-priority sync of scrollback, it couldn't. The only way to update the scrollback is to write the scrollback before writing the current screen.)
There could be a happy medium mode or setting that enables scrolling as you described by emulating the keyboard behavior, but mosh would recognize through some buffer capacity if you catted more than about 1MB and skip to printing the last 0.75MB of it. (Makes me want to go on and read that patch you sent to see if I can get it to work like that...)
You probably didn't mean to read more than about 1000 lines of text into your terminal so you could mouse cursor select it by hand into your paste-buffer. You answered very informatively, thank you.
(I haven't really contributed to mosh and haven't looked at the codebase in years, so asking on the mailing list or IRC is probably a better place to start)
Tip for getting scrolling back also working:
Ctrl+a Esc let's you scroll through your screen (with iTerm also with your mouse scroll).