Video playback timing is actually pretty difficult, with only trade-offs, no "right" solution.

The easiest solution is to use your computer's accurate clock to time the frames. This means, if your video is at 30 FPS, you will have gone through exactly 3000 frames after 100 seconds. But your monitor might not be refreshing at exactly 60 FPS, it might be targetting the 59.94 standard, or it might be targetting 60 FPS but with a clock that's a bit less accurate than your computer's, so every now and then, you get a stutter as a video frame has to shown for 1 refresh interval longer or shorter than it should.

What we could do instead is to calculate a fixed pattern to display frames. If your monitor is 30Hz and the video is 30 FPS, we show exactly one video frame for every monitor refresh. If your monitor is 60Hz and the video is 30 FPS, show each video frame for 2 monitor refreshes. If the monitor is 30Hz and the video is 60 FPS, skip every other video frame. This will already introduce some deviation from the "ideal" timing if your display clock isn't super accurate. And if the monitor is 59.94Hz, that's probably close enough to 60 that we want to show each frame of 30 FPS video exactly twice, but that means the video is playing at 99.9% of its intended speed.

10 seconds over 20 minutes is a surprisingly big gap though. Playing every frame of 30 FPS video twice on a 59.94Hz monitor would net a 1.2 second deviation over 20 minutes.

EDIT: Why the downvotes? Everything here is correct according to my understanding of the situation after implementing video playback software and working with other people's. If anything is wrong, please correct me, don't just silently downvote.

Usually playback will be locked to a hardware clock, typically the audio clock. This does mean that the audio clock and frame clock will drift over time, as they're not locked to each other; it doesn't matter how accurate the clocks are, they will drift. This is one source of dropped/doubled frames. It's possible to sync to the video framerate instead, of course, though audio desync artifacts tend to be more noticeable and objectionable than video ones. You can handle this by dynamically measuring the clock drift and variably resampling the audio, but this is relatively complicated and computationally expensive. Generally this type of drift isn't too noticeable, these clocks are usually well within 100ppm, which would result in 1 frame slip every few minutes.

The issue with framerate mismatch is a fundamentally different one, in that it's not a synchronization problem. It's the result of a non-integral ratio between display frames and playback frames, and exists regardless of clock source. So it's the harder of the two to deal with, and is worse in magnitude too at several slipped frames per minute with e.g. 29.976fps vs. 30fps. In the classical approach, this simply results in frames getting doubled or skipped, since it's not tracking the video rate at all, it just asks for the display to be updated at its frame-time, and the display system will either get a chance to show it or not. The slightly more refined approach is to play back at the closest integral ratio (e.g. play 29.976fps video at 30fps on a 60fps display), and resample the audio to match this new rate, but again this is computationally expensive and complicated.

mpv team has a good writeup: https://github.com/mpv-player/mpv/wiki/Display-synchronizati...

Excuse my ignorance, but don't monitors take their clock signal from the host? I would think that synchronizing a stream of frames to the display refresh rate is a mismatch to handle on the display end, not the host end where frames might get doubled or skipped (and eventually introduce long-term timing errors). This is assuming an 60Hz rate advertised by the monitor.

I don't know whether the clock is technically in the host machine or in the monitor, I imagine it depends on the type of connection. But the idea of a different "display clock" and "playback clock" still applies, since a program's own timekeeping based on usleep + gettimeofday will be different from the monitor clock, wherever that happens to reside.

And unless you're using some kind of variable refresh rate, the monitor always, always refreshes on a fixed interval. It doesn't do any "synchronizing a stream of frames to the display refresh rate". But with VSync, you can synchronize your software with the display refreshes, which is how you can decide to draw every frame twice and that sort of stuff.

Pixel clock is generated on the host. All the VSYNC/HSYNC pulse timings are also generated by the host. Monitor has little choice but to respect it, or refuse to operate.

On some embedded platforms, if you can't get a precise clock you need from clock subystem, or massage the display mode timing to fit whatever pixel clock you can generate, you may be running your display at 34 FPS instead of 60 FPS if you're not careful eneough. And you'll barely notice unless you measure it, or try to play video or do anything else sensitive to timing.

Don't forget that while you might not notice a 20 minute video taking 1.2 seconds more than 20 minutes - but you will notice if the audio plays at the correct speed and ends after exactly 20 minutes - and you will probably notice before the video ends.

Yeah, you obviously have to sync audio and video with each other. It's just usually not that important to sync the audio and video with some "true" time as measured by an atomic clock.

> That said, I've had daily issues with it, from it slowing video playback (weirdly, by about 10 seconds over 20 minutes or so, only noticable when watching something synchronized with someone else), over crackling noises with speakers when no audio is playing, to straight up lags and skips every few seconds with certain devices.

How is that "better than pulse ever did"? The only issue I had with PulseAudio on several machines in last decade or so was Bluetooth A2DP connection being very capricious.

I have recently switched to PipeWire and it actually improved Bluetooth, but I had to disable Wireplumber's libcamera backend because it made it regularly eat 100% of RAM. No complaints after doing that. Crackling noises or desynchronized video playback would instantly take it into "unacceptable" territory.

Sounds like you have severe issues with both PA and PW. Maybe your hardware is the problem?

I estimate about 75% chance it's a configuration problem on his end.

The "I hate Pulse Audio and especially Lennart" crowd tends to want to either go back to "Alsa" or tried at some point to install the old OSS drivers and then gave up.

This means trying to configure dmix to work properly, trying to configure your audio outputs with tools like 'alsamixer' and things of that nature.

This is about a 100% guaranteed approach to screwing up audio in Linux. For years it is what unsuspecting newbies were told by "really smart people" on the internet when faced when any sort of audio issue and all it really does is ensure that their OS audio is going to turn into a dumpster fire.

The only way to fix it is to aggressively find your "custom" audio settings for every application, find all the alsa configuration files and delete them. And then find out where your distribution saves your alsa mixer settings between reboots and link them to /dev/null and reboot.

Audio settings of this nature are very persistent and nothing Pulseaudio or Pipewire can do to help you fix it.

As someone using strictly alsa/dmix/alsamixer for decades now, I'm surprised to learn my OS audio is a dumpster fire.

To me it's a low-feature deterministic stack that rarely does what I don't want it to.

But I haven't introduced a bunch of fuel for the fire like bluetooth audio devices either... to each their own.

> The "I hate Pulse Audio and especially Lennart" crowd tends to want to either go back to "Alsa" or tried at some point to install the old OSS drivers and then gave up.

Well, yeah, since when pulse showed up and started breaking people's audio it effectively replaced alsa (even though they kind of operate at different layers). So if you didn't like that, of course the solution was to rip out Poettering's bugfest and revert to alsa. Thankfully at some point pulse stabilized a lot and in my experience tends to just work out of the box these days, but people tend to have their habits set by the early versions that were pushed out before they were ready.

> How is that "better than pulse ever did"? The only issue I had with PulseAudio on several machines in last decade or so

It's "better than Pulse ever did" because they do not have the same hardware as you, and because whatever they had before was worse.

Lesson 2 about discussing technical issues on the internet: other posters almost certainly do not have the same hardware as you.

(this is also relevant when it comes to discussing software performance - the fact that Slack loads in 5 seconds on your desktop does not make it "fast", because the entire internet does not have your desktop)

> Lesson 2 about discussing technical issues on the internet: other posters almost certainly do not have the same hardware as you.

Thanks for reiterating my point, I guess.

> Crackling noises or desynchronized video playback would instantly take it into "unacceptable" territory.

> Sounds like you have severe issues with both PA and PW. Maybe your hardware is the problem?

Seconding this! My experience with PA and PW has been basically identical to yours. If possible, GP, try some different hardware! You shouldn't have to put up with that crap.

Hm, I might try that. I swapped headphones, mics, etc, the next thing to swap would be the motherboard, or to get a dedicated sound card.

> How is that "better than pulse ever did"?

Pulse would cause games to crash or not even start (i know this because switching to pipewire resolved that), for example. Pulse, for me, was beyond broken on every machine I used it, in some capacity. PipeWire is, too, but in a way that doesnt ruin my day ;)

> Maybe your hardware is the problem?

Sure, could be, though my hardware is quite common.

If you haven't, post an issue on the Pipewire repo. I've found that the main author, Wim Taymans, is very proactive in fixing any issue if they're reproducible.

I've opened two bugs, and two times he fixed them personally in less than a week, which is incredible for an open source project of that size.

I've had the same experience. Had a couple issues and they ended up fixed in the next version.

Not to defend Pulse too hard but I use Linux every day and am specifically interested in audio and outside low latency work (where I use jack), Pulse has rarely been a problem since many many years. What exactly are the issues people experience?

I regularly hit a couple.

This one breaks audio and video in virtual machines:

https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/46...

The effects of this one vary:

https://gitlab.freedesktop.org/pipewire/wireplumber/-/merge_...

For me it breaks audio in Chrome and VLC:

https://github.com/wwmm/easyeffects/issues/1789

I guess I wasn't very clear.

It was issues with Pulse I wanted to know more about. Low latency is obviously not it's forté, otherwise it just works, for me that is.

You were pretty clear. I just completely misread. :)

>> Audio on linux is a complete joke

Yes, for example, the zoom app is now one of the most widely relied on apps for connecting people over networks. To get its audio output to fairly represent the input (e.g. for musical content), the input client is supposed to use the 'original audio' setting. That setting is not yet available on the linux version of zoom. On the output end, the audio is a mess, sounding as if somewhere along the way a handshake has turned into a fistfight.

Audio is complicated. Another example: an ordinary amateur, part-time, casual guitarist might nowadays play with thousands of dollars of pedals and processors plugged into his signal path. He gets this to work by plugging and unplugging those devices and their cables in various configurations until everything sounds best. Try providing that level of flexibility in software with reasonable default settings for all of the myriad of applications and environments in which one might try to apply computer audio.

By the time pulse finally worked right!

2060: PipeWire11 is being deprecated and replaced with a compositing audio server named Weyband

I don't even know why this is even necessary. Back in the day I just set up the ALSA daemon to save/restore volume levels and that was it. In all programs I select ALSA as output and boom done. So I'm confused to why pulse audio and pipewire even exist.

Raw ALSA can only have one program playing sound unless your hardware has support for hardware mixing (this would generally mean it's very old and also very expensive). More likely you are/were using ALSA's dmix plugin which is a small barebones version of the core functionality of pulseaudio or pipewire. They replace dmix but also give you things like per-program volume control, better latency, better power usage, better mixing, bluetooth support, the ability to do fancy things with audio routing without restarting every program playing audio, etc.

I am not sure about "better latency". The latency with Pulseaudio cannot be controlled via GUI and is often quite large (on order of hundred ms or more). In my opinion, to get minimum latency one needs to access audio card directly, without intermediate daemons.

PulseAudio optimizes for power consumption but tries to be adaptive to handle lower latency as well. PipeWire seems to optimize for latency but try to be adaptive to reduce power consumption as well. These are conflicting desires, lower latency requires higher power usage because you have to use shorter buffers instead of just giving the sound card a big chunk of audio to work through and putting the CPU to sleep. JACK could potentially still do better and ALSA could potentially still do better than even that but PipeWire's goal is to be close enough or even match JACK while still having all the other features people use PulseAudio for.

PA isn't great when it comes to latency, but JACK and PW are.

I'm not really sure what the difference between "ALSA just works" and "ALSA as configured by my distribution and including the DMIX plugin just works" is.

Like at the end of the day option 2 is still "ALSA just works".

Because the complexity of audio is hard. If you just have a desktop system with stereo speakers it is easy. However if you ever plug a set of headphones in things get hard. Should all sound now go through the headphones or not? Different people will want different answers (I have headphones plugged in all the time, but I'm not wearing them all the time, so if someone sends me an IM I need the beep to come out the regular speakers). There are monitors that have built in speakers making this harder, does the user have better speakers plugged into the computer output or not?

That is the regular desktop user. If you do any for of audio processing (pod casting, music recording) you may have a large number of different inputs and output connected, and want to do mixing of some sort. Likewise if you do videos you may have more than one "webcam" that you want to mix. These are fairly common use cases (many people have a dream of being a musician or an actor, and so have this type of setup that they are trying to use in the efforts to make it big)

> Should all sound now go through the headphones or not?

The correct answer to that specific question: as a sane default, all sound should now go through the headphones. That's a sane default on the Iphone, probably on Macbooks, and it also happens to be the one that at least a stock Gnome or Ubuntu desktop chooses. In fact I have trouble thinking of a saner default than that one!

Forgetting about sane defaults seems to be a common problem in FOSS communication. It's unfortunate, because describing something as unknowable can convince a new developer to avoid inspecting the code around that particular problem.

E.g., it took a long time for Gnome to get the sane default I describe above. I'd bet that lurking in the appropriate mailing list history is at least one didactic post from an audio developer explaining how we can never truly know what a user's preferred setup could be.

Because they do things many users find useful:

-Per-application volume control

-Seamless hardware switching mid-stream (plug in headphones)

-Different channel configurations into the same device simultaneously

-Cross-application volume ducking

Because ALSA doesn't allow two programs to use the same audio card simultaneously. ALSA is good for exclusive access, when you want to remove everything between an application and audio card.

Also ALSA cannot do other nice things, like switching audio output when you plug an USB audio card.

It can and does as it was configured by most distributions.

The dmix plugin does, not ALSA.

pulseaudio exists because it's user friendly.

JACK exists becase some people need low latency and dynamic control over non-trivial audio routing.

pipewire attempts to bring the best of both worlds and make it container friendly for security purposes.

> JACK exists becase some people need low latency

Wouldn't latency be lower if the application would access audio card directly without any daemons in between?

Then you would only be able to do fraction of functionality JACK provides you. May be enough for the most simple use-cases, but people who reach for JACK aren't usually after those.

The answer is, surpisingly maybe, no. JACK (and PipeWire) manage to add features without adding any latency to what ALSA can provide. Will it consume more CPU? yes, it does...

Most of the things people below say ALSA didn't have, OSSv4 actually had. But basically, they made it proprietary, Linux switched to ALSA, and when OSSv4 became open source again it lost traction. But from my experience at the time, it had none of the problems ALSA had, it had none of the problems pulseaudio had, and it had none of the problems ESD had.

This is a clear case of when ideology won over an objectively better product to the detriment of all users for more than a decade.

Ehhhh OSS is really not great, even v4. Re-using Unix file I/O functions for audio sounds clever, but falls apart pretty quickly in the real world. Even in the best case where you manage to shove every audio feature into open, read, write, and ioctl, it's never going to feel good to use (type safety...).

I don't mourn the loss of OSS.

OSS relies on mixing in the kernel, which is bad for reasons explained here: https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ...

Also with OSS you still can't seamlessly switch between devices, bluetooth or arbitrary signal routing between apps. So, no... good riddance.

Firefox can't do sound with raw ALSA. No, I know that doesn't explain why PulseAudio exists. But I think Pipewire can do more - isn't it supposed to be able to move video around as well as audio? Or something.

I also just use raw ALSA. It's there anyway, it works, and I don't need more daemons (I don't need NetworkManager either).

I've experienced that same lack of synchronization with other people.

I can recommend Syncplay. For a variety of reasons, but it’ll also deal with desync.

I don't think it works on Twitch...

* Audio on Linux "used" to be a complete joke. Pipewire fixed that.

For me, its still a WIP than a full fix, but definitely better than pulse :)