"Basically output the next N frames, given an input sample. It's trivial to estimate the error as you just split the video in two equal pieces. And try to guess the second half."

I actually did an experiment with something very close to this in the 90's, using simple morphing. For example, a video of a scene panning, but it was actually still photos with morphing between them to create the animation as more of a smooth pan instead of the seizure-porn strobe effect of flipping pictures.

It didn't work as well as I thought it would, but I think what came out of it was possibly a form of art. Everything looked almost normal, but there was a really frenetic halo that accompanied anything that moved on the screen - especially if it moved in any direction where parallax plays a part in understanding what went where. The wriggley halo artifacts looked cool, but maybe only for a rock video. There was none of the smoothness I expected or hoped for.