sniklaus

Video demo: https://twitter.com/simon_niklaus/status/1450863810160459777

That means that there were 175574 HTTP GET requests, but my guess is that if someone fast-forwards that that might trigger another request. So the actual number of views may be a little lower.

I see your point now. This post originally pointed to the teaser video and not the project website (I do not know who changed that, must have been an admin): http://content.sniklaus.com/kenburns/promo.mp4

That looks awesome, thank you for sharing!

Only very few people in comparison watched the full video. Here are the logs for the content server from yesterday 5 PM PST, covering the previous 24 hours (search for /kenburns/promo.mp4): https://pastebin.com/4zTxL2aw

Looks like there is another source of traffic at the moment that dwarfs HN and I do not know where it comes from: https://imgur.com/qok0Wmk

Thank you for sharing your thoughts! We designed the automatic camera path estimation to minimize the amount of disocclusion which indeed simplifies the problem. As you correctly pointed out, inpainting the background is an additional challenge and while we address it, the inpainted results sometimes lack texture.

Good suggestion, thank you!

It is the result of a lot of work (a lot) and I truly believe everyone in my situation with the right mindset would have been able to achieve it. Being a Ph.D. student and having a chance to do internships at companies like Adobe or Google does sometimes feel like a privilege though, so I understand that not many are in the same situation.

I just got the daily email from my server with some metrics. It has experienced over 500 gigabytes of traffic in the last few hours and the video is only 3 megabytes in size. I definitely did not expect such a HN-effect.

It seems like he is using them extensively but not too extreme. Looks great, thank you for sharing!

It's complicated but the gist of it is that they are trying to commercialize it, yes. For what it is worth, you might be able to find it in commercial software once they were successful with their business endeavors.

2. Yes and there are two mechanism for handling seams. First, the inpainting which extends the point cloud and can provide a higher sample rate. Second, a postprocessing step that heuristically fills in any seams that may still be present despite inpainting.

4. The downside of it is that one needs a lot of resources in order to try all of these variants, which not everyone is lucky enough to have access to.

Those masks look great, thank you for the pointer!

I unfortunately did not get the approval to release the context-aware frame interpolation, but one of my older works is open source for research purposes: https://github.com/sniklaus/pytorch-sepconv

1. I am not sure about the scale of the individual loss functions anymore, my apologies. I determined the combination of the two losses via a simple grid search and seeing what works best (plus / minus a magnitude did not make that much of a difference).

2. The points are just splatted to an image plane, more advanced point cloud rendering techniques would be better though. There is no video frame interpolation, each individual frame in the output video is a rendering of the point cloud from a different camera perspective.

3. I am not sure about multi-view stereo, COLMAP still seems like the state of the art for that. But neural networks definitely outperform classic techniques for single image depth estimation.

4. Common architectures just did not do as well as I was hoping for so I tried about 1500 model architectures. I started with an architecture that intuitively seemed right and then gradually explored / refined alterations of it. It ultimately was a lot of trial and error.

Since I have done the work while interning at Adobe, I am afraid that I will not be able to commercialize it myself due to patenting. I agree with you that it has great potential, there are still quite a few failure cases though and it is a little hit and miss.

There definitely is room for improvement. Our proposed framework uses Mask R-CNN for object segmentation. However, Mask R-CNN only as a resolution of 28x28 pixels for the object masks if I remember correctly (which is then resized to the actual resolution of the object in the image). So there are a lot of limitations from that alone.

Yeah, we just are good at seeing artifacts in human renderings, it can easily look wrong or uncanny. In contrast, if a boulder is rendered geometrically wrong we usually do not notice it.

I was thinking about what I should post and decided to make this little teaser since it demonstrates the gist of our work within a few seconds. I understand that the typically Hacker News audience may find the paper more worthwhile though.

I wholeheartedly agree with you and believe that depth perception is a key aspect that is is missing in the status quo of viewing still images!

The effect itself is actually not too uncommon. It is just very tedious and time-consuming to do manually since it requires segmenting the elements in the scene, arranging them in a 3D space, filling in any holes and specifying a virtual camera trajectory.

More or less. Estimating the depth from a single image is highly challenging and far from solved. We thus had to made sure that the depth estimate is suitable for synthesizing new views. And we are not explicitly modelling layers, we actually model the scene geometry as a point cloud. But you definitely got the gist of it. By the way, one advantage when estimating the scene geometry in a video is that it is possible to employ structure from motion.

Thank you for sharing your thoughts!

* There definitely should be more variety in terms of camera paths. Our framework either supports fully automatic results (as shown) or ones with a manual camera path. For an automatically generated slideshow, one should probably add more variety.

* There is some great work in this area, commonly focusing on portrait images. But yes, once the scene geometry has been estimated sufficiently well, one can definitely add some nice out-of-focus effects.

* That would be an interesting research direction. I am not an expert in relighting, but I can imagine that this requires a lot of work to make sure that the scene composition looks believable.

That is called guided upsampling and we are in fact doing just that to get a better estimate of the scene geometry. Specifically, we estimate the scene geometry at a resolution of 256x256 pixels (for a 1:1 aspect ratio, it varies for other resolutions) and then use the original image as a guide to upsample this estimate to 1024x1024 pixels. So yes, it could be used to a greater extent but it needs more work to be able to do this.

I just watched the trailer to that special, really interesting insights into the vision of sharks. And kudos on the nifty special effects to illustrate it!

That is a nice way of putting it, thank you for chiming in!

I would be happy if it gets its own name in the future. I am voting against "sniklaus" effect though since this was a team effort and I could not have done it without my mentors.

Thank you for clarifying, I see where you are coming from! My background is in research and it is fairly common to extend / reuse an existing terminology, but I totally understand and accept your point of view.

Our work focuses on physically correct depth. We have noticed that such effects created by professional artists emphasize the parallax to an extend that is not physically correct. As such, artists still seem to know best how to animate a catchy parallax effect.