No they have not made it worse. 1080p encoded at lower bitrate can look better than 720p at a higher bitrate. Or vice versa. Even to the trained eye. It depends on the scene dynamics.* In the ladders they posted (called a RD chart) we can clearly see they have made a better selection of the resolution to encode at a given target bitrate increase the video quality. They have posted the BD rates for this stuff, the gains are substantial (30% or more visual quality at the same bitrate).
> Also, something that's kind of weird about all this is that encoders (x264, x265, etc) already have their own rate control algorithms that decide on a frame or scene basis how many bits to use. Netflix taking an approach like this is equivalent to claiming that they're capable of doing a better job than these codecs in an automated way - so why not just contribute the code needed to achieve these improvements back to the projects they're using?
Because they've built a split and stitch encoder on top of the codecs. That means the code is built around x264, not into it.
> Last, it's very weird to select individual frames from an encode and not a selection of frames. The comparisons purport to show that the lower bitrate encode is better than the higher bitrate encode, but in fact (if I understand how they're using rate control correctly) what they are showing is a single frame in the lower bitrate encode that uses more bits than the same frame in the higher bitrate encode. So it's arguably not a fair comparison. And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
This is extremely true.
* High action scenes tend to benefit from lower resolution with more bits spent on inter compression while slower scenes benefit from tighter intra compression.
Well, of course that's correct. Take 1080p encoded at 100 Mbps and 720 encoded at 300 Mbps, I guarantee you that 1080p wins every time. :-)
> In the ladders they posted (called a RD chart) we can clearly see they have made a better selection of the resolution to encode at a given target bitrate increase the video quality.
I understand how an RD chart works, but I disagree with your conclusions. Reread my original post carefully - my point is that if you're stuck with 1080p as the maximum resolution, than the quality you can get with the revised ladder is strictly worse, because Netflix will want to jump up to UHD resolutions, but your device won't support it. (And even assuming that the UHD will be better based on this chart is kind of a leap, because it depends on what artifacts you're most sensitive to. Objective metrics ≠ subjective experience. In several of their comparisons, I prefer the lower res version because the new one is simply too artifacted.)
> Because they've built a split and stitch encoder on top of the codecs. That means the code is built around x264, not into it.
I understand the technical difference, but I'm still not convinced. Fundamentally what you get at the end of the day is a single continuous h.264 stream. The quantizer is chosen based on some rate control algorithm, and IPB frame choices are made by another algorithm. Netflix's improvement amounts to having a better rate control, that understands scene differences and uses lower quantizers on more complex scenes. There's fundamentally no reason why you couldn't add such a rate control algorithm to x264. It might require using a 2-pass mode, but it could be done.
"As a side note, we do have some additional points, not shown in the plots, that are used in resolution limited scenarios — such as a streaming session limited to 720p or 1080p highest encoding resolution. Such points lie under (or to the right of) the convex hull main ladder curve but allow quality to ramp up in resolution limited scenarios."
These are for their HEVC encodes. Apple (and most others) require H264 to have a separate ladder which will be 1080p limited and will be generated on different encode set and maximum video quality.* Almost any HEVC capable device can go to 4k, even if the screen is only 1080p. It will be used than down scaled.
> (And even assuming that the UHD will be better based on this chart is kind of a leap, because it depends on what artifacts you're most sensitive to. Objective metrics ≠ subjective experience. In several of their comparisons, I prefer the lower res version because the new one is simply too artifacted.)
Certainly objective and subjective metrics have differences, but VMAF does correlate pretty tightly with subjective metrics. This is a very detailed subject, I summed up many of my thoughts on the differences in my demuxed talk last year here: https://www.youtube.com/watch?v=nCUsXhSPyyw
* https://developer.apple.com/documentation/http_live_streamin... 1.23 iirc
Really? I wasn't aware Netflix even offered anything other than H.264 at <= 1080p. I know Youtube doesn't. If this is all HEVC maybe you're right and the resolution issue isn't such a big deal. If this is actually the case I wish they would have said so explicitly in the blog post.
> VMAF does correlate pretty tightly with subjective metrics. This is a very detailed subject, I summed up many of my thoughts on the differences in my demuxed talk last year here: https://www.youtube.com/watch?v=nCUsXhSPyyw
Thanks, I'll check that out when I have the time. I'd still emphasize that it's quite possible for objective metrics to be fooled (that's why codecs have psy tunings!), and I do prefer several "old-ladder" images over the newer versions in Netflix comparisons here. I suppose I've become rather cynical about claims made on the basis of metrics ("now 50% less bitrate for the same quality!"), when to my eye, web video has always been very bad and is getting worse as companies keep reducing the bitrate.
[youtube] dQw4w9WgXcQ: Downloading webpage
[info] Available formats for dQw4w9WgXcQ:
format code extension resolution note
249 webm audio only tiny 49k , opus @ 50k (48000Hz), 1.18MiB
250 webm audio only tiny 65k , opus @ 70k (48000Hz), 1.55MiB
140 m4a audio only tiny 130k , m4a_dash container, mp4a.40.2@128k (44100Hz), 3.27MiB
251 webm audio only tiny 136k , opus @160k (48000Hz), 3.28MiB
394 mp4 256x144 144p 73k , av01.0.00M.08, 25fps, video only, 1.72MiB
278 webm 256x144 144p 97k , webm container, vp9, 25fps, video only, 2.25MiB
160 mp4 256x144 144p 107k , avc1.4d400c, 25fps, video only, 2.05MiB
395 mp4 426x240 240p 159k , av01.0.00M.08, 25fps, video only, 3.42MiB
242 webm 426x240 240p 217k , vp9, 25fps, video only, 4.04MiB
133 mp4 426x240 240p 290k , avc1.4d4015, 25fps, video only, 4.48MiB
396 mp4 640x360 360p 340k , av01.0.01M.08, 25fps, video only, 6.68MiB
243 webm 640x360 360p 396k , vp9, 25fps, video only, 6.96MiB
134 mp4 640x360 360p 484k , avc1.4d401e, 25fps, video only, 8.27MiB
244 webm 854x480 480p 586k , vp9, 25fps, video only, 10.03MiB
397 mp4 854x480 480p 603k , av01.0.04M.08, 25fps, video only, 11.37MiB
135 mp4 854x480 480p 741k , avc1.4d401e, 25fps, video only, 11.56MiB
247 webm 1280x720 720p 1035k , vp9, 25fps, video only, 17.67MiB
136 mp4 1280x720 720p 1077k , avc1.4d401f, 25fps, video only, 16.72MiB
398 mp4 1280x720 720p 1133k , av01.0.05M.08, 25fps, video only, 22.07MiB
399 mp4 1920x1080 1080p 2106k , av01.0.08M.08, 25fps, video only, 40.74MiB
248 webm 1920x1080 1080p 2666k , vp9, 25fps, video only, 58.46MiB
137 mp4 1920x1080 1080p 4640k , avc1.640028, 25fps, video only, 78.96MiB
18 mp4 640x360 360p 601k , avc1.42001E, 25fps, mp4a.40.2@ 96k (44100Hz), 15.19MiB (best)You can confirm this by loading a video, right clicking somewhere in the player and opening 'stats for nerds'
No it doesn't. Depending on the distance from the screen and indeed the screen resolution, you won't benefit from the extra 1080p resolution, but you could benefit from the extra bits. 1080p60 is 3.7GBit uncompressed with 10 bit 444, so that's 37:1 compression. 720p60 is 1.7gbit, or 5.5:1 compression, so far fewer compression artifacts.
And the graph you posted isn't quite showing what you're saying it is: these show the "4K" curves. When you're on a 1080-max device but have a strong connection, there are higher-bitrate 1080 streams available; they're just not on this chart because that's not what it's showing.
In other words, what the curves are showing is just that they're switching to higher resolutions at lower bitrates than before, when your device supports them. Really the way to read it would be drawing lines vertically: given a fixed bandwidth, the "optimized" encode is always giving higher quality and often higher resolution.
Maybe they will? But it matters most for 4K performance, so that's where they started?
> encoders (x264, x265, etc) already have their own rate control algorithms
But they simply operate on a stream without taking future frames into account. This does a first entire pass on the film holistically to determine where keyframes should go and settings per-shot. It can't be backported to codecs because they work linearly.
> And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
The artifacts look worse only because it's zoomed in so crazy far. On a cinema screen, the additional sharpness will be clear, the artifacts not so much.
Given that x264 already has a 2 pass mode, I don't see why that is necessarily the case. Even CRF mode uses mbtree by default, which is a pretty complicated rate control algorithm. x264 also has pretty intelligent keyframe determination, I almost always see I frames on scene transitions.
> On a cinema screen, the additional sharpness will be clear, the artifacts not so much.
I think it's probably the other way around. The closer you are to a screen, the more likely you are to notice increased resolution. But if there are large patches of the image full of artifacts, you'll be likely to see that even far away.
x264 will not align keyframes across resolutions and encodes. In addition the pershot encodes optimize more than just idr frame placement. They also optimize other encoder parameters such as aq.
> I think it's probably the other way around. The closer you are to a screen, the more likely you are to notice increased resolution. But if there are large patches of the image full of artifacts, you'll be likely to see that even far away.
That isn't how this works. It's not a strict tradeoff between more artifacts and less sharpness by changing resolution. Downscaling -> Upscaling is simply another form of lossy compression. It may look better or worse than using those bits in another spot.
Sure, that's true. (Though I don't know why it matters that keyframes aren't aligned.) But at the end of the day the point is that Netflix has a better rate control algorithm, and this could be built into x264, even if it might require a significant amount of work. (Which I'm sure the x264 developers would be willing to do for a substantial quality improvement.)
> That isn't how this works. It's not a strict tradeoff between more artifacts and less sharpness by changing resolution.
Of course it's not a strict tradeoff. It's a loose one. And yes, it may look better or worse. That's really my only point in that section of the comment: that bumping up the resolution earlier in the ladder as they're doing is not a pure win, and it may look worse to some people depending on their viewing conditions.
You can't ABR adapt without aligned GOP boundaries.
> Of course it's not a strict tradeoff. It's a loose one. And yes, it may look better or worse. That's really my only point in that section of the comment: that bumping up the resolution earlier in the ladder as they're doing is not a pure win, and it may look worse to some people depending on their viewing conditions.
Of course not, that's why the perform an analysis of both options and select the better one. That's what the algorithm does...
Yes, that's true of course, but not really relevant to whether you could port Netflix's work on scene-adaptive rate control to x264. Maybe you'd lose aligned GOPs... but for a lot of purposes (offline?) that doesn't matter.
> Of course not, that's why the perform an analysis of both options and select the better one. That's what the algorithm does...
The only point I've ever tried to make on this subject is that in some cases this approach fails. "Perform an analysis" is such a high level description that it misses the fact that this is being done according to some objective metric that may disagree with an individual viewer's personal preferences or viewing environment. In fact, just because an objective metric says 4k > 1080p doesn't mean the difference will be noticeable at the viewing distance the viewer is at, whereas the additional artifacts introduced by moving from 1080 -> 4k without a significant bitrate increase may very well be visible!
The best way to understand why this is used is to think of a movie — when there are shots that are totally, absolutely black, like scene changes, normal 1-pass CBR encoding uses the exact same amount of data to that part as it uses for complex action scene. But by using VBR and multipass, encoder “knows” that this piece is OK with lower bitrate and that bitrate can be then used for more complex scenes, thus creating better quality for those scenes that require more bitrate.
Also, you're making a slight mistake. About a decade ago (?), x264 was changed so that even 2-pass mode is really just using CRF under the hood. It's not "achieving the best quality", it's just figuring out what quantizers you need to hit your average bitrate target exactly. Some more information about that here: https://trac.ffmpeg.org/wiki/Encode/H.264
> normal 1-pass CBR encoding
You may be confusing constant bitrate (which no one should ever use) with CRF (constant rate factor), which varies the bitrate from frame to frame and scene to scene, without having to do a 2 pass encode.
This really is their core business and somewhere they can have a competitive edge, though. Even if you think it's unethical or that it might make business sense to contribute back to the upstream projects, you can see why they might want to keep some of that secret sauce in-house.
Is that still the case? They switched from x265 to beamr for HEVC. Not sure if they are still on x264 on recent encodes.
Also, something that's kind of weird about all this is that encoders (x264, x265, etc) already have their own rate control algorithms that decide on a frame or scene basis how many bits to use. Netflix taking an approach like this is equivalent to claiming that they're capable of doing a better job than these codecs in an automated way - so why not just contribute the code needed to achieve these improvements back to the projects they're using?
Last, it's very weird to select individual frames from an encode and not a selection of frames. The comparisons purport to show that the lower bitrate encode is better than the higher bitrate encode, but in fact (if I understand how they're using rate control correctly) what they are showing is a single frame in the lower bitrate encode that uses more bits than the same frame in the higher bitrate encode. So it's arguably not a fair comparison. And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
Edit: here is an example of what I mean when I said that they may have made the quality at 1080p worse. If what you end up watching is the highest bitrate 1080p available, you would get higher quality on the old latter than on the newer one. https://i.imgur.com/rzPR7Sh.jpg Actually, on some of the examples, even 720p on the older ladder is better than 1080p on the newer one!