The X protocol is extremely chatty, does not compress images, and uses TCP. It's virtually impossible to make any sort of latency guarantees with that sort of protocol. Pipewire can use several compressed formats and the OP used UDP to stream the media off Pipewire to the remote display.

Yeah, I know running X11 over the wire doesn't hold up, but that's not what's being done here with Wayland either; since we're locally grabbing pixels off the virtual display and then flinging them over the wire via custom UDP protocol, X11 vs pipewire only matters for the initial handover.

The compositor may be able to transfer image data to PipeWire in zero-copy fashion, which may then be consumed by hardware video encoder without any copy on the path. I'm not sure whether you'd be able to do that using Xorg as display server.

Actually it is possible to do low latency while using X server (for those not on Wayland), but it seems much faster to stream a video of the screen area than to use X protocol directly (which I have often used over SSH in the past to control another desktop system) - Xpra seems much faster to me. I've tried x11vnc too. Xpra is cool. Just my subjective experience, but proper benchmark stats would be cool to see.