except not everyone has decided to use that easing of the things. it seems like they go out of their way to make things difficult
My favorite is in how MP4s are not all the same. A video captured on a DSLR or other acquisition camera source might have +/- in the video/audio sync when imported into an NLE. Yes, the container utilizes PTS syncing to playback the video/audio streams in sync, but this +/- slop in sync is annoying AF to professionals capable of seeing the sync slop. Average consumers probably never notice. I thank all the gawds in all the heavens that I no longer have to edit MP4 source captured content and all of the manual slipping in the NLE.
But...sometimes you want one particular thing, that thing is relatively simple (compared to e.g. H.264 decoding), and you have needs ffmpeg doesn't meet. Then you probably can do better. A couple examples that have come up for me:
* specialized .mp4 muxing. My NVR project does this in a way where it can do HTTP range serving, so put the moov up front, produce the length quickly, and produce a given byte range without iterating through all the bytes before. ffmpeg can't do that, understandably. Hard to do in a general-purpose library; the container muxing input interface doesn't expect all the necessary information is available up front. But it's not that bad as a one-off. As you say, the consensus around a small number of containers really helps.
* RTSP. Consistent with your "modules that mostly work, usually, mostly", ffmpeg has various long-standing bugs in its RTSP handling (e.g. https://trac.ffmpeg.org/ticket/5018), has the risk associated with protocol handling in a non-memory-safe language, doesn't have much of a test framework for this, etc. This problem is relatively approachable (though hardly trivial), so I had success making my own library.
There are infamously a lot of tasks that "look easy upfront but have a lot of underlying complexity" and ffmpeg is not one of them. It's a task that looks hard upfront and then when you get down into the weeds it's an eye-bleeding eldritch monstrosity.
The consensus around MP4 and MKV containers and some relatively small number of codecs (H264, H265, VP1, AV1, etc) has substantially eased things but video codecs, audio codecs, and container files are just complicated af.
And believe it or not - this is still way, way better than it used to be! Flaskmpeg was never stable, Virtualdub was nice but development ended with avi container files, there were some forks that added various features, etc, but ffmpeg and OBS Studio both actually just work incredibly well compared to what came before.