I agree, the improved version is worse. At first glance, it looks better than the original, but then numerous artifacts become apparent. For example, check out the little girl's face. It's jumbled in the "improved" version, but very clear in the original.

It looks like most of the perceived improvement comes simply from the upgrade to 60fps, which is achieved trivially through interpolation.

I wonder if the video would have gotten as much attention had AI not been mentioned.

In most ways, yes. For me the first 5 seconds or so were fantastic. It felt like seeing a behind the scenes "How it was made" documentary where I was seeing the actors out of character. Or like it removed the sense of being from another era. Soon the artifacts and warping became distracting, and then watching the original put it back in time. Maybe it was a sense of nostalgia but overall the original seemed better to me.

Someone posted a colorized[0] version and it had somewhat "jumpy" colors that change over time. I think one problem is you need temporal consistency for this to really work properly. If you pick a color for an object, or you upscale an object a certain way, it has to stay that way over a potentially unlimited number of frames. That's already hard to achieve because neural networks are typically bad at long-term dependency, but it's even more difficult if the object moves, because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. At the moment, neural networks don't have sophisticated models of the world.

[0]: https://www.youtube.com/watch?v=EqbOhqXHL7E

>...because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with.

I agree with most of what you said, but disagree with the need to know underlying fundamentals: most 5 year olds can throw and catch a ball without needing to understand calculus or Newton's laws of gravity.

they don't need to know the formulas or mathematical logic behind physics, but 5 year olds do have a working model of physics in their head. And AI wouldn't need to know the actual formulas either, but it would need some working model that owuld allow it to differentiate between those

That's right. It has to understand how objects move, and the boundaries between objects, what's part of a given object and what isn't, as well as how a given object will be affected by light when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it.

> It has to understand how objects move[1], and the boundaries between objects[2], what's part of a given object and what isn't[3], as well as how a given object will be affected by light[4] when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it.

Each one of the items you've listed has been severally accomplished by ML to varying levels of success (baseline being "workable", and more progress is being made.) I probably would have found better references had I spent longer than 5 minutes searching

1. https://www.youtube.com/watch?v=AGm3hF_BlYM

2. https://www.theverge.com/2019/12/9/20999646/google-arcore-au...

3. Same as 2 :-). AR Occlusion requires both. Additionally many recent phones use ML models to fake depth-of-field; boundary between foreground objects has to be detected

4. Also see 2; Augmented Reality is hard. bonus: https://www.awn.com/animationworld/how-klaus-uniquely-combin...

I work in deep learning research. In my opinion, pose estimation with one human centered in a frame is nowhere near the level of difficulty of constructing an accurate model of a complex a scene with multiple people and moving objects of various sizes. For instance, it wouldn't be so hard to detect that there is a train in this video, but to do upscaling correctly, you need a 3D model of this specific train. Yes, there exist deep learning models that try to do 3D reconstruction, and so far, I haven't seen anything that works robustly. Combining this information with an upscaling model is another challenge. It's not as easy as just plugging components together.