ssh carthoris ls /mnt/media/Movies | grep Spider

(this is just an example). Note that in this example, we have two processes running on two different machines. Indeed, the OSs and systems on these machines may be, um... different. Indeed, I routinely include "cloud" machines in pipelines. Indeed, with ssh, the -Y (or -X) option can introduce a GUI to a part of the command.

I have wished that shar was part of SUS. Also, I find that "exodus" is useful (across Linux anyway -- the systems have to be "reasonably" homogenous). https://github.com/intoli/exodus

In order for this to work over SSH, the SSH client and server would need to be enhanced to exchange this data, and also an SSH protocol extension would need to be defined to convey it across the network.

One might define IOCTLs that work on pipes and PTYs to send/receive control messages. So sshd would read control messages from the PTY and pass them over the network, and the SSH client would receive them and then pass them on to its own stdout using the same ICOTLs. (Alternatively, one might expand the existing control message support that recvmsg/sendmsg supply on sockets to work on pipes and ptys as well.)

Any program supporting such an out-of-band signalling mechanism would have to gracefully degrade when it is absent. If your SSH client or server, or some program in your pipeline, or your terminal emulator, etc, doesn't support them, just fall back on the same mechanisms used today to determine output/input formats.

(rsh is such a deprecated protocol, there would be no point in trying to extend it to support something like this.)

Ha ha, I had dreamed up something like this in one of my wilder imaginings, a while ago: A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline :) Don't actually know if the idea makes sense or would be useful.

And another apparent one is image viewer or editor.

There seems to be some precedent for this sort of thing. For example, DVTM can invoke a text editor as a "filter", where the editor UI is drawn on stderr and result saved to stdout.

I did some experimenting recently and found you can open a new /dev/tty file descriptor and tell curses to use this, the rest of the application can continue reading stdin and writing stdout as normal.