The examples in the video were automatically generated without user feedback. It is actually also possible to manually specify the camera path in order to achieve the effect that you are expecting, for more information feel free to have a look at the video on the project website.

Understood. This is an area of image research I have an interest and some experience in (image segmentation, background inpainting, etc., through machine learning.)

I think your results are wonderful. I was just pointing out that from my background (a television producer - most recently Shark Week, etc.) that the Ken Burns effect and parallax effects are considered two different concepts (that can be combined, but mean different things.)

I did not mean to shoot you down or anything, I in fact greatly appreciate your feedback! I unfortunately do not have a television / movie producer background but wish to learn more to better understand and support the workflow of professionals in this area. On a different note, I have recently not had time to watch television and hence just looked Shark Week up. It looks popular, congrats!

The grantparent is right when it comes to terms:

- Ken Burns effect -> zoom and pan around the image.

The Ken Burns effect is a type of panning and zooming effect used in video production from still imagery. The name derives from extensive use of the technique by American documentarian Ken Burns. (Wikipedia)

- 2.5D parallax effect -> what we see in the link

https://blog.pond5.com/16853-create-2-5d-parallax-effect-ima...

https://www.youtube.com/watch?v=WcSdhtxId2o

Note that these two can (and frequently are) be combined, and Ken Burns himself does so occasionally.

I would like to add that our method estimates the scene in full 3D space. Have a look at the additional results, there are a few results that exemplify this when the geometry is a little more complex: http://sniklaus.com/papers/kenburns-results

I'd like add that, naming aside, your algorithm results looks amazing, and you could even have a business (or a successful app) here.

I know a few online services that do manual (cutting in photoshop, setting on a 3D plane, etc) still photo to 2.5D parallax, e.g.:

https://sundstedt.se/blog/services/2-5d-parallax-3d-effect-a...

there are Fiverr gigs doing it, and tons of filmmakers (me included when I dabble with that), do it ourselves too (which takes time in Photoshop, Premiere, etc) from time to time (for news, documentaries, and such).

In fact I am about to pay a good 100-200 to have 4-5 photos done this way for a project...

Since I have done the work while interning at Adobe, I am afraid that I will not be able to commercialize it myself due to patenting. I agree with you that it has great potential, there are still quite a few failure cases though and it is a little hit and miss.

My special was “Andrew Mayne: Ghost Diver” where I built a suit to fool great white senses. Not shown in the special was the work we did trying to understand how sharks process visual data differently than humans. One interesting note is that they don’t seem to do image segmentation like we do (people in a shark cage are just a noisy box-thing and not discreet objects.)

Again, your project is great. I’ve been trying to find a simple workflow to do something like this (currently it’s remove.bg and OpenCV - with a lot of tinkering.)

I just watched the trailer to that special, really interesting insights into the vision of sharks. And kudos on the nifty special effects to illustrate it!

> Not shown in the special was the work we did trying to understand how sharks process visual data differently than humans.

I would absolutely love to read that paper if it existed. Also, congratulations on the special!

I think OP meant for the 3D to be a qualifier of the Ken Burns effect. Meaning he added parallax to the Ken Burns effect to produce a "3D Ken Burns Effect".

I don't have any experience in any of it but I'm just offering another explanation of the discussion.

Here's the page and video explaining it: http://sniklaus.com/papers/kenburns

It might be a nice side result to use your tech to identify foreground elements automatically to generate the camera motion, without using the 2.5D layering. Then you’d have a more traditional Ken Burns effect. It might not work at picking a specific face out of a crowd, but it would work for a lot of scenarios.

I find Apple’s automatic Ken Burns effect to be way too noisy and almost useless because it zooms in/out from random places in the pictures. When I manually edit a Ken Burns effect myself, there’s a lot less movement, and the movement is so much more meaningful and relevant to the video.

Thank you for your feedback! I highly value the opinion of working professionals like you and am taking notes. :)

Yes. This is amazing technology and looks fantastic.

No it is not what is generally referred to as a ‘Ken Burns’ effect, which I think is a distraction in the title.

Because this is awesome!

Also how can I use this for a project I am literally doing production on right now??

Edit: People usually just call this parallax.

Perhaps it would also be worth mentioning Ken Burns and his documentaries.

They're absolutely worth watching and gives BBC a run for its money.

Perhaps start with The West and then move on to Civil War. My personal favorite was perhaps the one on Jazz, made in 2001.

His Vietnam series was great.

Civil War... I guess I already knew a lot about it, so what I noticed mostly was more sympathy than I'd like for a sort of Lost Cause nostalgia, and not much talk of slavery. But I guess he had to sell it for its time...

Loved the Vietnam series too. Have watched it a couple of times on Netflix.

I'm 40 yo European so I knew close to nothing about the Vietnam war other than what I had seen in some movies. Very in depth and emotionally powerful. It also seemed pretty objective to me as a complete outsider.

That's because he had hacks like Shelby Foote trying to convince people that the Civil War wasn't caused by slavery.

The Central Park 5 is utterly heartbreaking. I think it's Ken Burns' closest to today. I believe all its main people are alive today.

One is in the White House. What he did in this case was unconscionable.

https://www.pbs.org/kenburns/the-central-park-five

His brother Ric Burns made a documentary about the Donner Party. It really is a masterpiece of non-fiction horror.

Unfortunately it's unavailable for streaming. Every time someone uploads it, it gets taken down.

This Peabody Award clip is all that's available. The DVD can be had for cheap on Amazon.

https://youtu.be/w-eTjCuNDoE

https://film.avclub.com/the-most-terrifying-pbs-special-of-a...

Baseball and National Parks are two of my personal favourites.

His Vietnam is particularly powerful since it's filled with personal accounts.

A new one starts tonight on Country music. Even though I'm not a big fan of the genre, I expect it to be fascinating.

The Roosevelts was my intro to Ken Burns and it was the only TV show ever that I could not stop watching. Since then, I've also been very impressed by the quality of all the other Ken Burns documentaries.

I'm not sure that they give the BBC a run for its money - you're talking about quite a lot of Beeb output ;-) I agree that they are of an unusually high standard.

BBC documentaries, many of them atleast are having a reputation beyond what they deserve I think, often times the real information in them is sparse, and they present overtly simplistic narratives, with too much padding.

In comparison to what, though? It's easy to knock the Beeb, but compare the output (TV, Radio, and Web) to anything vaguely similar and you will probably appreciate the quality. They're not just making docus, it's so much more. Funnily enough, the Beeb aired the Burns Vietnam, too.

For example compared with the work of Asif Kapadia in recent years, which almost entirely avoids voice over narratives.

I watch episodes 5-8 of Jazz like once a year

Those are great documentaries....

I also really enjoyed Empire of the Air, covering the history of radio.

Looking at the website of OP, they agree with you in the classic definition of Ken Burns effect, and call this "3D Ken Burns".

We were doing a lot of reading to better understand video producers an their usage of the Ken Burns effect. We are ultimately still research scientists with no hands-on video-production experience but we made sure to do our homework. :)

Some thoughts based on looking at the video a few times more (disclaimer - I'm not a video professional)

* There needs to be some variation in the zooming if you use this in a slideshow. I'd even say bias it towards zooming out more often, since that gradually reveals more of the image.

* You might have enough information to work with here to develop a similar "focus pull effect from single image" algo. That would be really cool.

* Maybe also you could try to develop a "sunset fade" for images where you detect blue sky. Graduallu adding a gradient yellow-orange fade for the sky, gradually warmer and darker foreground and non-sky background.

Thank you for sharing your thoughts!

* There definitely should be more variety in terms of camera paths. Our framework either supports fully automatic results (as shown) or ones with a manual camera path. For an automatically generated slideshow, one should probably add more variety.

* There is some great work in this area, commonly focusing on portrait images. But yes, once the scene geometry has been estimated sufficiently well, one can definitely add some nice out-of-focus effects.

* That would be an interesting research direction. I am not an expert in relighting, but I can imagine that this requires a lot of work to make sure that the scene composition looks believable.

As for 2, Focos app has recently added the fake depth blur, and it is both absolutely impressive (oh my! It gets most of the image so right) and discernible (we are so sensitive about the visual that one wrongly estimated region - often a 'hole' like the ground between the legs - is enough to ruin the magic).

http://focos.me/

That's the way I took it as well. Of course, I'm definitely not a movie producer or a videographer (is that what they're called?)

Amazing but yea, not the Ken Burns effect.

MacOS has "Ken Burns" screen saver if you are on Mac. There's also plenty of examples of "Ken Burns" effect on youtube

https://www.youtube.com/watch?v=WjdWvmjgBa0

Also Wikipedia

https://en.wikipedia.org/wiki/Ken_Burns_effect

The authors might want to change the description on their amazing effects if they want to avoid lots of threads and comments about it not being the Ken Burns effect (or maybe that will generate more comments)

I am confused, a quote from Wikipedia: "The Ken Burns effect is a type of panning and zooming effect used in video production from still imagery."

Our framework takes a single still image and animates it via panning and zooming while adding 3D parallax. Does adding the 3D parallax not make it the Ken Burns effect anymore? Please let me know in case I am misunderstanding anything.

Yes, the examples that we have shown do not zoom as much as common Ken Burns examples. This stems from our framework processing the input at low resolution due to limitations in deep learning and the overall complexity of the problem. As such, there is not enough detail that could be zoomed into. This will be improved in later generations.

> Does adding the 3D parallax not make it the Ken Burns effect anymore?

Yes.

Specifically, the Ken Burns effect describes taking a still, 2D image and panning and zooming on the 2D image in a motion picture.

Adding 3D parallax makes it something quite different in the eyes of many of us on this forum. It is cool! We would just not use the words "Ken Burns effect" to describe what you've made.

On the contrary I think it basically serves the same purpose as the Ken Burns effect, except it tastefully incorporates the natural depth of the image.

One might even argue Ken Burns would have paired the panning and zooming with this parallax effect if he could.

I look forward to this enhancing basically all photo slideshows for a long time to come.

That is a nice way of putting it, thank you for chiming in!

I know nothing about this. But it seems obvious that, whatever we call your work, Ken Burns' use of panning and zooming is a subset of it.

Thank you for clarifying, I see where you are coming from! My background is in research and it is fairly common to extend / reuse an existing terminology, but I totally understand and accept your point of view.

There is probably a name for this kind of "taking a 2D image, separating it into 2d cutouts, and panning with parallax". It's an effect that's been around for a while (hand done, not automated as you have it) but it's never been referred to as a "Ken Burns Effect"

Googling "How to make a parallax video from 2d pictures" brings up lots of tutorials, none of which mentions "Ken Burns Effect"

https://www.google.com/search?q=how+to+make+parallax+video+f...

Conversly searching for "How to make the Ken Burns Effect" will show that it's not the same effect your software makes

https://www.google.com/search?q=how+make+the+ken+burns+effec...

I think adding the 3D parallax gives the footage a very unique effect, and quite different from the traditional Ken Burns 2D scale and pan. I would say enough to make it deserve its own name, such as the sniklaus effect. Ken Burns is Ken Burns. This is something quite different, even though it may be inspired by KB.

I would be happy if it gets its own name in the future. I am voting against "sniklaus" effect though since this was a team effort and I could not have done it without my mentors.

Would it not be possible to do the image processing on the original image to create several pixel masks which then draw in the high resolution image data?

For instance, you take the source image at resolution n*n and do all processing at n/2 or n/4. Then at the moment you are going to composite the finished image you instead draw "pixels" from a source image which contains X,Y not RGB. Then you upscale the output image x2 (or x4) and replace each X,Y index with the relevant pixel from the source image (adding X%2,Y%2 or X%4 to the indexes to return to source resolution).

That is called guided upsampling and we are in fact doing just that to get a better estimate of the scene geometry. Specifically, we estimate the scene geometry at a resolution of 256x256 pixels (for a 1:1 aspect ratio, it varies for other resolutions) and then use the original image as a guide to upsample this estimate to 1024x1024 pixels. So yes, it could be used to a greater extent but it needs more work to be able to do this.

Would Ken Burns have some input into whether his name is used for this?

It's called, "using a rostrum camera".

https://en.wikipedia.org/wiki/Rostrum_camera

If you grew up watching PBS documentaries there was always a rostrum camera operator somewhere in the credits.

In the UK, Ken Morse was/is best known for this.

https://twitter.com/kenmorsecredits

Pretty much what I thought as well. But cool nonetheless.