When the application host is really resource-starved and does not need to supply much content to a renderer, we are better off using more abstract network interfaces. On one hand, HTML+CSS+js are the remote rendering protocol to the browser which becomes the remote renderer. On the other hand, sensor/data interfaces become the remote data source for a GUI application that can be hosted somewhere more appropriate than on the resource-starved embedded device.
What I will miss from SSH X-forwarding is not remote rendering. It is per-window remote display. I would be perfectly happy with remote applications doing their rendering within their own host, and just sending their window contents to my display server, where my local compositor puts the window content onto my screen. Protocols more like VNC or RDP could be used for each of these windows, adaptively switching between damage-based blitting, lossless and lossy image codecs, or even streaming video codecs. What is needed is the remote window-manager and display protocol to securely negotiate and multiplex multiple windows opening and closing rather than one whole desktop.
What I won't miss from SSH X-forwarding is window and application lifecycle tied to the network connection. Let me disconnect and reconnect to my remote application and its windows. Let me share such a connection to have the same windows displayed on my desktop and my laptop, or to window-share with a coworker down the hall...
Unix is different things to different people and to remove the network capability from X is a step back, not a step forward. There are a whole pile of neat things that you can do with networked graphics, so much, in fact that Plan 9, in many ways a successor of Unix used that theme as the core of many of the system services. All of them work locally as well as remote.
Hardly any of the use cases I have heard for networked graphics actually require or even care where the rendering occurs. They are just concerned with application software installed and running in one place causing pixels to appear for a user elsewhere. These could work just fine with remote display streams, with all rendering happening on the same host where the application executes.
I ponder what it is that I really dislike about VNC or RDP, and I find it is the nested desktop and lack of integrated window management. I think I'd be fine with some kind of "rootless" variant to let me SSH-forward a set of windows to a Wayland compositor. I would prefer it to have more flavor of tmux/screen to let me connect and disconnect, and easily define named sessions or groups of windows that I would want to grab together.
They haven’t removed it from X have they? You can keep running X if you want to (you may need to start maintaining it yourself.)
They don’t use it so they don’t want to implement or maintain it. Seems super reasonable to me?
There's also remoteapp which allows you to just run one program remotely, but haven't got much experience with this.
Last time I checked it was a X extension, if at all.
What does an optimization have to with anything? So what if X does not have a native GPGPU driver, that's really more a function of hardware manufacturers support than anything else. And if you are not too picky about all your stuff being 'open' then NVidia's X driver and CUDA on linux co-exist just fine with accelerated graphics and GPGPU support.
RDP is more closely related to VNC and something like 'screen' than X, which is more like a networked resource that you can access through remote clients.
RDP is an application sharing protocol, X is a screen sharing protocol, the two are radically different from each other which gives each advantages and disadvantages that the other lacks and a bunch of overlap. Most notably, with X security was bolted on as an afterthought. RDP, originally reading the screen contents and sending those over (compressed) does not allow for much interaction with what gets sent over, it is rather low level whereas X sends over display primitives. Again, X's initial - rather naive - implementation made a ton of assumptions because the original implementors had nice fat workstations and fast networks to work with and so the real-world utility of these features was rather limited.
Anyway, you could write books about this and not get all the details, besides the various implementations of both but all that there is to this is that due to their history and intended application the two are very different beasts and X offers a versatility that not a whole lot of people need but when you need that versatility you need it badly.
Per a discussion the other day, my RPis are in a drawer. Have we swapped out screwdrivers for RPis in the modern junk drawer?
No, they've been added to the screwdrivers. I've found it challenging to use RPi to drive screws.
The Unix philosophy: do one thing, and do it well. If you want a remote renderer, write a piece of software that does just that, specifically.
Yes, this means that unlike many C-S architectures, the server is the part nearest the user and the client (may be) further away. But each did its respective job and stuck to its knitting.
Keep in mind that when X11 was first created, and for much of its early life, through the 1990s, it was used for remotely accessing another (or multiple other) systems. The networking capability was baked-in. Nevermind that security was bolted on later, here, have a MAGIC_COOKIE....
You're trying to argue that there should be two distinct pieces of software, a local and a remote renderer, for simplicity.
Rendering does not fit in that space well. The physical resource of a display and graphics card really only allows one renderer to run, so it's not really useful to say "just write two renderers, one local and one remote".
The unix philosophy is not a good argument. It's a principal by which you can make decisions, but it doesn't always lead to good decisions and it's quite open to interpretation as well.
Nowadays, perhaps, but that's software bloat in action, software getting slower faster than computers gaining in speed. In the last millennium if one had a login one could connect to the computer in Daresbury, UK that had a copy of the Cambridge Structural Database and search crystal structures. Many people at British universities did that. It would render graphics on an X server. Once I tried it on holiday at the East Coast of the US, and it wasn't a painful experience at all. Then again, it was the old version that still had PLUTO, which was a program from the computing stone ages.
X forwarding goes the complete opposite direction to pipes the whole X protocol over the network. Convenient -- yes, also dead simple to implement -- yes, actually works -- well no. You see the X ecosystem also assumes all rendering is local, that fact is just hidden in edge cases. For example Nvidia graphics work on X with an extension that both has a client and server component -- as soon as you forward X traffic that's looking for the server-side extension... crash.
RDP by contrast does the right thing(tm) but also the hard thing by defining an intermediate object model which is semantic enough to kill the need to send pixels over the wire most of the time but flexible enough to allow for it when necessary.
I'd agree with you for modern applications that are not well written, I've been running X applications over the network however for 20 years or so, and outside of things like.. well 3D applications, network X works pretty well - my only real complaint was not including sound too.
I used it in the early 90's to support the Unix/X11/TCL/Tk multi player version of SimCity, with a scriptable TCL/Tk audio server that could drive either /dev/audio on Sun/SGI/etc or NetAudio on NCD X Terminals. Other TCL/Tk clients like SimCity (but possibly others) communicated with it via the TCL/Tk "send" command, which bounced messages off the X server.
http://www.art.net/~hopkins/Don/simcity/simcity-announcement...
NetAudio had its problems, but basically worked. Except that it insanely mixed audio by AVERAGING (add waveforms then divide by the number of sounds) instead of adding and clipping, and I couldn't convince the NCD engineer that was the wrong way to mix sound, and to just add and clip instead, like sounds mix in the real world: When you talk over music, the music volume and the volume of your voice doesn't magically lower by half. Hopefully they've fixed that problem by now.
https://donhopkins.com/home/archive/NeWS/sound-mixers.txt
It evolved into the Network Audio System (NAS):
http://www.radscan.com/nas.html
I can't say how popular it is, or if anyone actually uses it, though:
The Release version is now 1.9.4 (10/07/2013).
Electron is panned by many these days for requiring an expensive network system for a local process. Why doesn’t X11 get the same scorn? Have we forgotten the days when a Mac could run a GUI in 128K while a Unix system struggled to run x11 (with no clients) in 4MB? X is heavy.
That's because running over the network didn't require ping-pong context switching back and forth between the X11 server and the client at every keystroke, so both client and server could run smoothly without getting switched out.
The X protocol is so chatty and ping-pongy that it could require several context switches per keystroke to handle the event and update the screen, when running both server and client locally!
If you actually properly understood both the argument you're making and the reason why most people dislike Electron and friends, then you'd not be making that argument. I'll give you the benefit of the doubt and assume you're joking.
It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.
Mac OS/X and iOS IOSurface:
https://developer.apple.com/documentation/iosurface?language...
http://neugierig.org/software/chromium/notes/2010/08/mac-acc...
https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...
Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...
https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...
https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...
https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
Care to elaborate, please?
Do you know if X11, like Electron, supports shared memory textures in the GPU like IOSurface and GL_TEXTURE_EXTERNAL_OES are for?
I can't force someone to understand something that they do not want to. More than enough people have written, much more eloquently than I, about why electron is bad. None of the critiques I've seen are 'because it relies on an expensive network system'.
EDIT: Hell, even if we assume that the critiques are about "expensive networked system"s, Electron is still incomparable to Xorg so the root-parent's post is making a false comparison.
What do you think of products like Google Stadia? https://store.google.com/us/product/stadia_founders_edition?...
This Google Stadia is using a remote display protocol. All the rendering is happening in the machine in the datacenter, where the CPU and GPU lives and the game executes. It's a bit like someone sitting in the datacenter playing the game while you live-stream it at home, except they also pass through input controls so you can steer the game from home instead of just watching it.
A remote rendering protocol, which is no longer really sensible, would try to keep the GPU in your living room while running the game in the datacenter. Instead of streaming the screen updates at a nice steady pace based on your display resolution, it would have to stream the drawing commands from the datacenter to your GPU, adjusting the geometry as the game simulation updates the world model and stalling with nice loading screens every time it needs to shuffle in different texture and model data from the game computer to your GPU via the internet.
The idea that a desktop compositor should be network aware by default is not really a defensible design decision. Why should the compositor be responsible for sending a compressed stream of OpenGl instructions over a network? This is an activity that is Someone Else's responsibility; making a graphics developer responsible for decisions about latency and compression is just going to reduce the number of people who can understand the codebase.
The reason for X was that in 2008 the driver manufacturers wouldn't support anything else. AMD and Intel have open source drivers now, so the reason isn't good enough to justify X as a near universal standard.
At 18m30, he mentions that everyone uses SHM and DRI2, which don't work over the network. He mentions X isn't "network transparent" anymore, it's "network capable". I'm not 100% sure what the difference is tbh.
At 40m10, he mentions that rendering on the "server" then compressing it to transfer over the network to display locally is basically VNC.
Of course, there are immense problems with this. For one thing, it's usually crippled by latency. Even if the client X application is in a Docker container on the same machine, applications are only halfway usable. Use a local Ethernet network, and things start to fall apart. Going over the Internet isn't really worth trying.
The other major problem is that there's no security. If you have an open X server, anyone can connect to it and run random applications on your display. But in addition to opening you up to annoyance from cyber-hooligans, allowing an X application to run gives it access to all your keystrokes, which may not be something you want. Beyond that, all the X data is going out over the network unencrypted. There's just no security model here.
That said, it would be cool if there were some kind of plugin for Wayland that would let you use this feature (or even something better, if someone wanted to implement that). I've wondered---can you set up XWayland to be an open X server and just let random X applications connect to it? It would be interesting to hear from Wayland experts on this point.
That said:
- X is burdened by this abstraction layer for networking
- Since people are using ssh as an app to do it, a case could be made that the networking is not really necessary and a dedicated graphics specific app might work just as well.
What I think you are describing is closer to what Wayland plus a remote display protocol might enable. Let the app use its beefy GPGPU hardware on the server and just blast the 2D frames to the workstation on each OpenGL buffer-swap, to be composited into the local framebuffer that drives the actual display. All the expensive texture-mapping, geometry calculation, and pixel shading would execute on the remote server.
No, I did not mean that.
You have mentioned multiple times terrible performance and latency, but I feel this is a strawman argument. People are streaming HD and 4K video over the internet all the time now and watching live-streamed video games, where someone is playing a GPU-intensive application and having the output encoded to an efficient video compression stream in real-time. These compression algorithms are also being used in practice for video teleconferencing solutions all over the place, by many vendors. The latencies are low enough to have normal conversation, movement and speech. They work reasonably well over consumer-level, low performance WAN connections and can work flawlessly over faster WAN or LAN paths. I don't understand how this contemporary scene suggest remote display is impractical.
Also, please don't get caught up in the VNC-style remote desktop metaphor as the only way to achieve remote display. An X-over-SSH styled remote protocol could just as easily be designed to launch applications on a remote host, where it is able to allocate server-side GPU hardware resources and run a renderer off-screen which buffer-swaps directly into a video codec as the sink for new frames, pushing those compressed frames to the user-facing system where they are displayed as the content of one application window. There is no need to conflate remote-display with having an actual screen or desktop session active on the host running the application. This is much like SSH itself giving us any number of pseudo TTY shell sessions without requiring any real serial port or console activity on the remote server.
If you didn't mean that the "CUDA" hardware in the server is doing rendering (like in Google Stadia) then I am guessing you mean that the application is doing CUDA data processing as part of its application logic before generating separate display commands to a remote X server to actually visualize the results. I am having a really hard time understanding why anybody would want to split and distribute an application like this, because every such data-intensive visualization I am aware of has much tighter coupling between those application logic bits and the renderer than it does between the renderer's output buffers and the photon-emitting display screen. This is why there are extensions, e.g. in OpenCL and OpenGL, to share GPGPU buffers between compute and rendering worlds. The connection between the application data-processing and the rendering can be kept local to the GPGPU hardware and never even traverse the PCIe bus in the host. Massive data-reduction usually happens during rendering (projection, cropping, occlusion, and resampling) and the output imagery is much smaller than the renderer state updates and draw-commands which contribute to a frame.
For what it's worth, I do know of cases where remote display is impractical due to latency, but these are also cases where remote rendering are impractical. For example, an immersive head-mounted display may require a local renderer to have extremely low latency for head-tracking sensors to feed back into the renderer as camera geometry so rendered imagery matches head movement. A remote rendering protocol would be just as impractical here, because the rendering command stream that embodies this updated camera and frame refresh would also have too much latency to accomplish the task. The only practical way to distribute such an application would be to introduce an application-specific split between the local immersive rendering application, placed near the user, and other less latency-sensitive simulation and world-modeling which can be offloaded to the remote server. A rendering protocol like X doesn't provide the right semantics for the asynchronous communications that have to happen between these two stages. Instead, you need many of the features reflected in a distributed, multiplayer game: synchronizing protocols to exchange agent/object behavior updates, local predictive methods to update the local renderer's model in the absence of timely global updates, and a data-library interface to discover and load portions of the remote simulation/model into the renderer on demand based on user activity and resource constraints in the local rendering application.
In order to justify network rendering at the windowing system level, I think you need a model that could reasonably give good performance to both local and remote applications without application developers having to care.
Sun's early windowing system was NeWS, which had a PostScript interpreter running on the display terminal. I think it was the better idea in the long run, but at the time X was easier to work with and wasn't patent encumbered...
It's not as good as a local display for resource intensive stuff, e.g. games, but X never was either (it was abysmal, in fact). It's a hard problem, but given that price/performance of computational power has mostly run ahead of network speed for most of computing history, and latency is a (mostly) insoluble problem for long distance networking, Web Assembly is often a better solution for pushing even resource intensive applications out to the edges.
It totally depends on X. I don't even know how to run wayland.
I am surprised that X was / is so under funded and under supported.
The other approach is (and that was like in the older days), when you avoid any tunneling but let X clients connect to open X servers across the network, like if your X display server listens on all NICs for clients. This setup is still technically possible but pretty much discouraged for security concerns. But with this setup you could also run a local xterm or whatever X client against a remote X server (like your neighbours ;), assumed it would accept your connection.
Like: DISPLAY=yourneighbour.host:0 xterm
BR, garbeam
Thin terminals brought this to the logical conclusion of not needing a desktop at all, but unfortunately X's protocol is too chatty and sensitive to latency to really shine across a WAN. These days you can work around all of that by opening a session via Guacamole or the like but that's still one tab in a browser per graphical session. There's no just using SSH in a for loop and opening up a raft of xterms to a bunch of machines in Wayland, AFAIK.
From the Wayland FAQ
Is Wayland network transparent / does it support remote rendering?
No, that is outside the scope of Wayland. To support remote rendering you need to define a rendering API, which is something I've been very careful to avoid doing. The reason Wayland is so simple and feasible at all is that I'm sidestepping this big task and pushing it to the clients. It's an interesting challenge, a very big task and it's hard to get right, but essentially orthogonal to what Wayland tries to achieve.