I remember sponsoring a feature in x264 (the best software encoder) and the level of complexity in that code is mind bending. The patch was to allow better CPU utilization when running with over 16 cores - something the authors hadn't run into at the time and had no systems to test on. I did a simple profile during encode, sent them the trace, in 2 days the patch was tested and working and it hit main in a week's time. Boom, all 48 cores used.
Made me realize just how much talent is behind these technologies and I still shake my head at how much money is generated with this and how little of that makes it's way back to the developers at the core of it.
If it's any consolation, not much more makes it to the talented developers in commercial products, either. It mainly ends up going to executives and sales.
I think it's primarily developers in other countries[1] that are getting severely underpaid. Hopefully that changes in the coming years.
[1] The most egregious example might be India. Around 7 years ago, I was on a plane from Kerala (a state in India), and the guy sitting next to me happened to run a software consulting company (in Kerala). His customers were from Japan and some other countries (can't remember), and his company wrote Linux drivers for proprietary hardware. He hired kids right out of college. I asked him how much he paid them, and he said without blinking, Rs. 15000 per month. Type in "INR 15000 to USD" to Google, and you're in for a shock. It's around $2600 per year, but probably closer to $3000 per year at the exchange rates back then.
American developers doing the same job could earn up to 100 times that, and at the very least 40 times that amount. I can't understand why on Earth salaries are 40 to a 100 times higher for the same exact work, and for work that could be done by anyone anywhere on the globe, but I hope this massive difference in pay converges and disappears soon.
I really really hope with the recent IPOS, all the devs that made good money end up doing startups that are more employee friendly than VC friendly.
But then, this is the ugly world we live in. The selfish gene yo!
Was it Dark Shikari? I remember there were lots of threading optimisation done, but that was... ~2010 time frame.
I really wish he is still working in the video encoding scene.
- once you have a reference implementation, you can have exact input/output pairs, which makes development easy, especially if the reference implementation exposes intermediate results.
- because of the previous point, testing is easy.
- even if you don't have a reference implementation, you can start by building a slow version of an encoder/decoder, generate testing data (intermediate outputs), and step by step optimize the system, while ensuring the system produces the same outputs at every step.
The problem is not in getting the output. The reference decoder decodes just fine. The difficult part is getting the output on time, e.g. getting you decoder to run fast.
The reference decoder has not been designed for speed, and getting things to run fast requires architecting everything very differently: it's not just a matter of applying micro-optimizations to reference code.
Also, the "intermediate results" might be different for each decoder architecture, so you can't really rely on the reference code for those.
So yes, while testing is indeed easy, writing a fast decoder is definitely not a "step by step optimization" process. I'd bet every performant decoder out there was rewritten almost completely at least twice.
On the other hand how encoder works is not specified at all apart from the obvious requirement that it has to produce something decodable by the above mentioned decoder.
However, it doesn't handle X.265 videos very well - pauses every few frames.
How come? I'm assuming its because 265 is much more CPU intensive.
What's the typical compression benefit of using x.265 vs x.264?
-----------------------------------
Update: noticed this comment below from @Causality1 "devices from over a decade ago have hardware h264 decoders. Extremely low powered modern devices can play very large 265 videos, if they have a hardware decoder."
Couple generations ago, AMD did the other way, they used GPU shader ALUs for video encoding/decoding. They stopped doing that because power efficiency is worse than custom silicon that’s now in all modern GPUs, even in mobile SoCs.
Yes, there are many software decoders out there, usually specialized for certain architectures (the team I worked with wrote a decoder for Texas Instruments DSPs, for example). Then there are hardware implementations, which are rarely just "hardware", it's usually hardware-assisted software that does the decoding.
Another tidbit of (potentially) interesting information: it took many years for H.264 to achieve its full potential, for two reasons:
1) Implementing a decoder is really, really complicated, especially if you want it to run fast, and if you think a miracle "hardware decoder" solves the problem, you need to remember that H.264 places heavy requirements on memory bandwidth and caching (B frames), so it's not like you can just feed a stream through a chip and get video frames on the other side.
2) Implementing an encoder is really, really complicated :-) — and there are many ways to encode a video stream. H.264 delivers the tools, but does not specify how you should use them. It took many years for decoders to get to a state where most of H.264 is used efficiently.