I wonder if instead of heuristics, machine learning could be used to figure out the best parameters.

In a somewhat-related topic, you might be interested in DLSS [0], where machine learning is being used in graphics rendering in games to draw the games at a lower resolution, then upscale the image to the monitor's resolution using a neural network to fill in the missing data. I imagine a similar thing could be done with video rendering, though you'd need some crazy computing power to train the neural network for each video, just like DLSS requires training for each game.

[0] https://en.wikipedia.org/wiki/Deep_learning_super_sampling

That seems likely at least.

You could actually use ML for all of the video decoding, but that research is still in it's early stages. It has been done rather well with still images [1], so I'm sure it'll eventually be done with video too.

Those ML techniques are still a little slow and require large networks (the one in [1] decodes to PNG at 0.7 megapixels/s and its network is 726MB) so more optimizations will be needed before they can see any real-world use.

[1] https://hific.github.io/ HN thread: https://news.ycombinator.com/item?id=23652753

That's already done today. Most modern codecs support variable bitrate encoding so more of the data budget can be given to high complexity scenes. Source video can also be processed multiple times with varied parameters and that output then compared structurally to the source to find the parameters that best encode the scene. This is beyond the more typical multi pass encoding where a first pass over the source just provides some encoding hints for the second pass. It takes several times longer to encode (though is embarrassingly parallel) but the output ends up higher quality for a given data rate than a more naïve approach.

Any apps you can recommend for giving this a spin?

x264 with 2-pass encoding? "Machine learning" doesn't only mean deep neural networks.

In the limit case, compression and AI are identical.

Once you get to an AI that has full comprehension of what humans perceive to be reality, you can just give them a rough outline of a story, add some information on casting, writers, and Spielberg's mood during production, and they'll fill in the (rather large) blanks.

That's a bit exaggerated, but I remember reading about one such algorithm a few days ago (by Netflix, maybe?). It was image compression that had internal representations such as "there is an oak tree on the left".

It would then run the "decompression", find the differences to the original, and add further hints where neccessary.

Sure. Machine Learning is just "heuristics we don't understand."

Or more typically Monte Carlo heuristics.

Reinforcement learning uses Monte Carlo a lot but traditional machine learning or deep learning don't.

I was hoping H266 was going to be a neural network based approach, but it looks like that might end up being H267.

Right now neural networks allow for higher compression for tailored content, so you need to ship a decoder with the video, or have several categories of decoders. The future is untold and it might end up not being done this way.

I think the number of previous frames for typical settings went from about 4 to about 6 as we went from H.264 to H.265. And the actual max in H.264 was 16. So that doesn't seem like a huge factor.