I just got the daily email from my server with some metrics. It has experienced over 500 gigabytes of traffic in the last few hours and the video is only 3 megabytes in size. I definitely did not expect such a HN-effect.

The video is 180 MB.

So 500 GB are about 2800 full video downloads.

Only very few people in comparison watched the full video. Here are the logs for the content server from yesterday 5 PM PST, covering the previous 24 hours (search for /kenburns/promo.mp4): https://pastebin.com/4zTxL2aw

I see your point now. This post originally pointed to the teaser video and not the project website (I do not know who changed that, must have been an admin): http://content.sniklaus.com/kenburns/promo.mp4

What does this line mean:

/kenburns/promo.mp4 | GET | 175574 | 33.4183748571803

That the video was watched 175574 times? That would indeed be a lot!

That means that there were 175574 HTTP GET requests, but my guess is that if someone fast-forwards that that might trigger another request. So the actual number of views may be a little lower.

Maybe upload the video to Youtube? The video is barely loading here.

Looks like there is another source of traffic at the moment that dwarfs HN and I do not know where it comes from: https://imgur.com/qok0Wmk

Probably reddit

Half of that must be me :-) This is so cool I watched it over and over again. Hope you'll be able to release the source - or otherwise a binary to play with. Would love to try this out on 3D rendered human anatomy from CT scans. Cinematic rendering there already looks amazing, with this it would really come to life (eg: https://www.ajronline.org/doi/10.2214/AJR.17.17850)

The results look impressive. Is the code or a tool available that would enable others to try it out on their images?

I may be wrong in thinking that given two researchers are working at Adobe, the plan probably is to keep it proprietary and put it in one of their software.

I am planning to release the code as well as the dataset but have yet to get approval to do so. If the work is done by an intern, then Adobe is usually fairly open in terms of sharing the work.

I'm doing a video slideshow for my niece's wedding in a couple weeks and would love to add this effect to a bunch of their photos (without spending a usual hours-and-hours in photoshop).

Hope you're able to release it ASAP!

Thank you, I'd really love to play with this!

I'd love to play around with it for a music video (non commercial)!

Thanks for clarifying. Looking forward to the digging into codebase.

Love your personal website.

Thank you for you kind words! I moved away from having separate websites for each project and am currently trying to bring all my projects to my personal website. I just added the Mini Chess game last night, which used to be an Android app.

Really impressive results. Is your "Context-aware Synthesis for Video Frame Interpolation" available as an implementation somewhere? It looks really good.

I skimmed the paper about the Ken Burns effect, and if you don't mind I have some questions. I hope I didn't miss the answers in the paper itself, I'll be sure to read it more carefully when time permits.

1. The loss function for the depth estimation is Ldepth=0.0001·Lord+Lgrad. Is Lgrad much bigger than Lord by design or will this basically make Lord tiny and almost unecessary? How did you arrive at this number and not a magnitude bigger or smaller?

2. How are you rendering the point cloud, like tiny discs in free space? When I think of point clouds I think of the typical LIDAR output renderings, but your results are continuous images, or video frames. Are the points rendered to an image plane and the result then interpolated?

And a couple more general ones:

1. Are all other depth estimation techniques other than NN obsolete now? Is there no point in estimating intrinsic camera parameters and epipolar lines when the state of the art seems to be to input one or more images into a NN and let it produce the depth for you?

2. How do you decide how the NN should look like, with various downsampling layers, convolutions, etc. I've seen people start with pretrained networks and retrain them, but how do you build a novel network based on the input and desired output?

I unfortunately did not get the approval to release the context-aware frame interpolation, but one of my older works is open source for research purposes: https://github.com/sniklaus/pytorch-sepconv

1. I am not sure about the scale of the individual loss functions anymore, my apologies. I determined the combination of the two losses via a simple grid search and seeing what works best (plus / minus a magnitude did not make that much of a difference).

2. The points are just splatted to an image plane, more advanced point cloud rendering techniques would be better though. There is no video frame interpolation, each individual frame in the output video is a rendering of the point cloud from a different camera perspective.

3. I am not sure about multi-view stereo, COLMAP still seems like the state of the art for that. But neural networks definitely outperform classic techniques for single image depth estimation.

4. Common architectures just did not do as well as I was hoping for so I tried about 1500 model architectures. I started with an architecture that intuitively seemed right and then gradually explored / refined alterations of it. It ultimately was a lot of trial and error.

Thanks for the reply.

Was it the university that didn't want to release it? Are they looking at commercializing it, or how does that work? Is it available in any commercial software? It kind of looks like magic and would probably be very useful for a lot of purposes.

2. So basically each point is projected to the image plane without perspective mapping? So in 3D, the further away from the camera they are the bigger they are so they all have the same size on the image? And that prevents any seams to occur in the pixel grid as things move around?

4. Experience, intuition, and elbow grease. Kind of what I thought, but I guess it's reassuring to see an expert in the field having to try 1500 variants.

It's complicated but the gist of it is that they are trying to commercialize it, yes. For what it is worth, you might be able to find it in commercial software once they were successful with their business endeavors.

2. Yes and there are two mechanism for handling seams. First, the inpainting which extends the point cloud and can provide a higher sample rate. Second, a postprocessing step that heuristically fills in any seams that may still be present despite inpainting.

4. The downside of it is that one needs a lot of resources in order to try all of these variants, which not everyone is lucky enough to have access to.