Re:PipeWire and fixing the Linux Video Capture stack
by PaulDavisThe1st on Sunday October 03, 2021 @12:01PM
(#28738167)
One clarification to TFA:
>And also we can have patchbay applications that supports video pipelines and not just audio, like Carla provides for Jack applications. To be clear this feature will not come for ‘free’ from Jack patchbays since Jack only does audio, but hopefully new PipeWire patchbays like Helvum can add video support.
There actually have been patches for JACK to enable it transport video just like it already handles audio & MIDI. We never accepted them into the mainstream.
The specific problem with video compared with audio is this: for audio, settling on 32 bit floating point as the canonical format for audio (and note, JACK did not prevent you from using other formats, even internally; people just didn't bother) was not even remotely controversial. Nothing is lost and much is gained via this choice.
For video, however, there are a plethora of possible format choices, each with their own distinct reasons for being, and often significant computational costs associated with conversion. This means that for video it's fairly problematic to define a single format for an entire workflow, which in turn means that figuring out when/where conversion between formats occurs is a significant task. For JACK, we decided not to try to do this, and left the system handling only audio & MIDI.
>And also we can have patchbay applications that supports video pipelines and not just audio, like Carla provides for Jack applications. To be clear this feature will not come for ‘free’ from Jack patchbays since Jack only does audio, but hopefully new PipeWire patchbays like Helvum can add video support.
There actually have been patches for JACK to enable it transport video just like it already handles audio & MIDI. We never accepted them into the mainstream.
The specific problem with video compared with audio is this: for audio, settling on 32 bit floating point as the canonical format for audio (and note, JACK did not prevent you from using other formats, even internally; people just didn't bother) was not even remotely controversial. Nothing is lost and much is gained via this choice.
For video, however, there are a plethora of possible format choices, each with their own distinct reasons for being, and often significant computational costs associated with conversion. This means that for video it's fairly problematic to define a single format for an entire workflow, which in turn means that figuring out when/where conversion between formats occurs is a significant task. For JACK, we decided not to try to do this, and left the system handling only audio & MIDI.