Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by metalrain on Wednesday April 24, 2024 @11:08AM
(#40145267)
Video quality seems really good, but limitations are quite restrictive "Our model encounters challenges when processing extremely long videos (e.g. 200 frames or more)".
I'd say most videos in practice are longer than 200 frames, so lot more research is still needed.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by danudey on Wednesday April 24, 2024 @02:16PM
(#40147782)
Sure, but that represents a lot of fast cuts balanced out by a selection of significantly longer cuts.
Also, it's less likely that you'd want to upscale a modern movie, which is more likely to be higher resolution already, as opposed to an older movie which was recorded on older media or encoded in a lower-resolution format.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by arghwhat on Wednesday April 24, 2024 @04:28PM
(#40149173)
I believe the relevant data point when considering applicability is the median shot length to give an idea of the length of the majority of shots, not the average.
It reminds me of the story about the Air Force making cockpits to fit the elusive average pilot, which in reality fit none of their pilots...
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by throwup238 on Wednesday April 24, 2024 @01:04PM
(#40146987)
The textures of objects need to maintain consistency across much larger time frames, especially at 4k where you can see the pores on someone's face in a closeup.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by vasco on Wednesday April 24, 2024 @02:02PM
(#40147651)
I'm sure if you really want to burn money on compute you can do some smart windowing in the processing and use it on overlapping chunks and do an OK job.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by mateo1 on Wednesday April 24, 2024 @01:31PM
(#40147305)
Huh, I thought this couldn't be true, but it is. The first time I noticed annoyingly fast cuts was World War Z, for me it was unwatchable with tons of shots around 1 second each.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by cubefox on Wednesday April 24, 2024 @04:43PM
(#40149355)
The first time I noticed how bad the fast cuts are we see in most movies was when I watched Children of Men by Alfonso Cuarón, who often uses very long takes for action scenes:
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by jonplackett on Wednesday April 24, 2024 @01:47PM
(#40147498)
So sad they didn’t keep to the idea of the book. Anyone who hasn’t read this book you should, it bares no resemblance to the movie aside from the name.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by danudey on Wednesday April 24, 2024 @02:17PM
(#40147795)
It's offtopic, but this is very good advice. As near as I can tell, there aren't any real similarities between the book and the movie; they're two separate zombie stories with the same name, and honestly I would recommend them both for wildly different reasons.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by KineticLensman on Wednesday April 24, 2024 @05:44PM
(#40150100)
It’s style is based on the oral history approach used by Studs Terkel to document aspects of WW2 - building a big picture by interleaving lots of individual interviews.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by btown on Thursday April 25, 2024 @12:25AM
(#40153408)
And similarly, I, Robot, which is much more enjoyable when you realize it started as an independent murder-mystery screenplay that had Asimov’s works shoehorned in when both rights were bought in quick succession. I love both the movie and the collection of short stories, for vastly different reasons.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by jonplackett on Wednesday April 24, 2024 @06:34PM
(#40150682)
The lost world is also a great book. It explores a lot of interesting stuff the film completely ignores. Like that the raptors are only rampaging monsters because they had no proper upbringing having been been born in the lab with no mama or papa raptor to teach them social skills
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by hobs on Wednesday April 24, 2024 @09:48PM
(#40152379)
Disagree, Jurassic Park was an amazing movie on multiple levels, the book was just differently good, and adapting it to film in the exact format would have been less interesting (though the ending was better in the book.)
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by jonplackett on Thursday April 25, 2024 @05:05AM
(#40155151)
I totally forgot the book ending! So much better.
I think like the motorcycle chase that they borrowed from the lost world in Jurassic world, they also have a scene with those tiny dinosaurs pecking someone to death.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by boogieknite on Wednesday April 24, 2024 @03:56PM
(#40148830)
Freal. To the degree that i compulsively count seconds on shots until a show/movie has a few shots over 9 seconds then they "earn my trust" and i can let it go. Im fine
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by jsheard on Wednesday April 24, 2024 @12:01PM
(#40145976)
Tale as old as time, in graphics papers it's "our technique achieves realtime speeds" and then 8 pages down they clarify that they mean 30fps at 640x480 on an RTX 4090.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by bmicraft on Wednesday April 24, 2024 @01:07PM
(#40147019)
You could probably mitigate this by using overlapping clips and fading between them. Pretty crude but could be close to unnoticeable, depending on how unstable the technique actually is.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by littlestymaar on Wednesday April 24, 2024 @12:26PM
(#40146337)
It's not so much that it would be impratical (video streaming, like HLS or MPEG-Dash, requires to chunk videos in pieces of roughly this size) but you'd lose the inter-frame consistency at segments boundaries, and I suspect the resulting video would be flickering at the transition.
It could work for TV or movies if done properly at the scene transition time though.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by anigbrowl on Wednesday April 24, 2024 @03:06PM
(#40148248)
Easily solved, just overlap by ~40 frames and fade the upscaled last frames of chunk A into the start of chunk B before processing. Editors do tricks like this all the time.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by kazinator on Wednesday April 24, 2024 @01:41PM
(#40147430)
Break into chunks that overlap by, say, a second, upscale separately and then blend to reduce sudden transitions in the generated details to gradual morphing.
The details changing every ten seconds or so is actually a good thing; the viewer is reminded that what they are seeing is not real, yet still enjoying a high resolution video full of high frequency content that their eyes crave.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by anigbrowl on Wednesday April 24, 2024 @03:02PM
(#40148202)
If you're using this for existing material you just cut into <=8 second chunks, no big deal. Could be an absolute boon for filmmakers, otoh a nightmare for privacy because this will be applied to surveillance footage.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by herculity275 on Wednesday April 24, 2024 @11:59AM
(#40145959)
Wonder how long until Hollywood CGI shops have these types of models running as part of their post-production pipeline. Big blockbusters often release with ridiculously broken CGI due to crunch (Black Panther's third act was notorious for looking like a retro video-game), adding some extra generative polish in those cases is a no-brainer.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by inhumantsar on Wednesday April 24, 2024 @12:28PM
(#40146376)
yeah and the only reason we don't see more of it was prohibitively expensive for all but basically Disney.
the compute budgets for basic run of the mill small screen 3D rendering and 2D compositing is already massive compared to most other businesses of a similar scale. the industry has been under paying their artists for decades too.
I'm willing to bet that as soon as unreal or adobe or whoever comes out with a stable diffusion like model that can be consistent across a feature length movie, they'll stop bothering with artists altogether.
why have an entire team of actual people in the loop when the director can just tell the model what they want to see? why shy away from revisions when the model can update colour grade or edit a character model throughout the entire film without needing to re-render?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by j45 on Wednesday April 24, 2024 @01:10PM
(#40147052)
Bingo. Except it looked like magic because the tech was so expensive and only available to them.
Limited access to the tech added some mystique to it too.
Just like digital cameras created a lot more average photographers, it pushed photography to a higher standard than just having access to expensive equipment.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by Version467 on Wednesday April 24, 2024 @03:31PM
(#40148546)
Hollywood has incredible financial and political power. And even if fully AI generated movies reach the same quality (both visually and story wise) as current ones, there’s a lot of value in the shared experience of watching the same movies as other people, that a complete collapse of the industry seems highly unlikely to me.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by rini17 on Wednesday April 24, 2024 @05:07PM
(#40149660)
What quality? Current industry movies are, for lack of better term, inbred. Sound too loud, washed out rigid color scheme, keeping attention of the audience captive at all costs. They already exclude large, more sensitive, part of population that hates all of this despite the shared experience. And AI is exceptionally good at further inbreeding to the extreme.
While of course it isn't impossible for any industry to reinvent itself, movie as an art form won't die....having doubts about where it's going.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by Version467 on Thursday April 25, 2024 @02:45PM
(#40161440)
I wouldn’t have any confidence in any predictions I make 100 years into the future even if we didn’t have the current AI developments.
With that said, I’m pretty confident that the movie industry will exist in 10 years (maybe heavily tranformed, but still existing and still pretty big). If it’s still a big part of current popculture by then (vs obviously on its way out) then I’d expect a collapse of it to require a change that is not a result of AI proliferation, but something else entirely.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by shepherdjerred on Thursday April 25, 2024 @03:32PM
(#40161980)
My point is that many talk about AI as though it's not going to evolve or get better. It's a mindset of "We don't need to talk about this because it won't happen tomorrow".
Realistically, AI being able to replace Hollywood is something that could happen in 20-50 years. That's within most people's lifetime.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by dguest on Wednesday April 24, 2024 @12:23PM
(#40146288)
I can't wait for the next explosion in "bigfoot" videos: wildlife on the moon, people hiding in shadows, plants, animals, and structures completely out of place.
The difference will be that this time the images will be crystal clear, just hallucinated by a neural network.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by dguest on Wednesday April 24, 2024 @12:19PM
(#40146240)
If you want to use your image for anything that needs to be factual (i.e. surveillance, science, automation) the up-scaling adds nothing---it's just guessing on what is probably there.
If you just want the picture to be pretty, this is probably cheaper than a bigger sensor.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by kfarr on Wednesday April 24, 2024 @12:01PM
(#40145991)
This is amazing and all but at what point do we reach the point of there is no more “real” data to infer from low resolution? In other words there are all sorts of information theory research on the amount of unique entropy on a given medium and even with compression there is a limit. How does that limit relate to work like this? Is there a point at which it can say we know it’s inventing things beyond x scaling constant because of information theory research?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by incorrecthorse on Wednesday April 24, 2024 @12:14PM
(#40146158)
I'm not sure information theory deals with this question.
Since this isn't lossless decompression, the point of having no "real" data is already reached. It _is_ inventing things, and the only relevant question is how plausible are the things being invented; in other words, if the video also existed in higher resolution, how close would it actually look like the inferred version. Seems obvious that this metric increases as a function of the amount of information from the source, but I would guess the exact relationship is a very open question.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by itishappy on Wednesday April 24, 2024 @12:34PM
(#40146494)
> This is amazing and all but at what point do we reach the point of there is no more “real” data to infer from low resolution?
The start point. Upscaling is by definition creating information where there wasn't any to begin with.
Nearest neighbor filtering is technically inventing information, it's just the dumbest possible approach. Bilinear filtering is slightly smarter. This approach tries to be smarter still by applying generative AI.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by thomastjeffery on Wednesday April 24, 2024 @01:03PM
(#40146975)
That point is the starting point.
There is plenty of real information: that's what the model is trained on. That information ceases to be real the moment it is used by a model to fill in the gaps of other real information. The result of this model is a facade, not real data.
They used digital upsampling techniques and colorization to make World War One footage into high resolution. Jackson would later do the same process for the 2021 series Get Back, upscaling 16mm footage of the Beatles taken in 1969: https://www.imdb.com/title/tt9735318/
Both of these are really impressive. They look like they were shot on high resolution film recently, instead of fifty or a hundred years ago. It appears that what Peter Jackson and his team did meticulously at great effort can now be automated.
Everyone should understand the limitations of this process. It can't magically extract details from images that aren't there. It is guessing and inventing details that don't really exist. As long as everyone understands this, it shouldn't be a problem. Like, we don't care that the cross-stitch on someone's shirt in the background doesn't match reality so long as it's not an important detail. But if you try to go Blade Runner/CSI and extract faces from reflections of background objects, you're asking for trouble.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by rowanG077 on Wednesday April 24, 2024 @11:29AM
(#40145557)
I am personally much more interested in frame rate upscalers. A proper 60Hz just looks much better then anything. Also would really, really like to see a proper 60Hz animate upscale. Anything in that space just sucks. But when in the rare cases it works it really looks next level.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by rowanG077 on Wednesday April 24, 2024 @08:31PM
(#40151722)
This just sounds like the AI is just not good enough yet. I mean it's pretty clear now that there is nothing stopping AI from producing close to or sometimes even exceeding human artists. A big problem here is good training material
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by rowanG077 on Thursday April 25, 2024 @12:39PM
(#40159741)
I didn't say that. I said that AI is capable of sometimes exceeding human artists. That is not the same thing as saying AI is exceeding the best human artist. If your training material is of high quality it's shouldn't be impossible to exceed human artists some or even most times, i.e. Produce better material then the average or good artists.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by duskwuff on Wednesday April 24, 2024 @06:22PM
(#40150562)
There are a lot of old porn videos out there which have become commercially worthless because they were recorded at low resolutions (e.g. 320x240 MPEG, VHS video, 8mm film, etc). Being able to upscale them to HD resolutions, at high enough quality that consumers are willing to pay for it, would be a big deal.
(It doesn't hurt that a few minor hallucinations aren't going to bother anyone.)
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by mlyle on Wednesday April 24, 2024 @11:45AM
(#40145792)
Reloading can get them in sync. But, it seems to stop playback of the "left" one if you drag the slider completely left, which makes it easy to get desynced again.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by turnsout on Wednesday April 24, 2024 @11:00AM
(#40145174)
I agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by jsheard on Wednesday April 24, 2024 @12:04PM
(#40146027)
Not really, DLAA and the current incarnation of DLSS are temporal techniques which integrate detail over multiple frames, using ML-guided heuristics. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets.
The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @11:07AM
(#40145257)
Why would you ever do that?
If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them.
Downscaling the images and then upscaling them is pure insanity when the high res images are available.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @11:34AM
(#40145636)
There is no use case, because it is a stupid idea. Downscaling then reconstructing is a stupid idea for exactly the same reasons why downscaling for compression is a bad idea.
The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @11:32AM
(#40145599)
That's absurd. I think anybody is aware that it is far superior to e.g. compress in the frequency domain than to down sample your image. If you don't believe me just compare a JPEG compressed image with the same image of the same size compressed with down sampling. You will notice a literal night and day difference.
Down sampling is a bad way to do compression. It makes no sense to do NN reconstruction on that if you could have compressed that image better and reconstructed from that data.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by acuozzo on Wednesday April 24, 2024 @12:10PM
(#40146109)
An image downscaled and then upscaled to its original size is effectively low-pass filtered where the degree of edge preservation is dictated by the kernel used in both cases.
Are you saying low-pass filtering is bad for compression?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by itishappy on Wednesday April 24, 2024 @07:13PM
(#40151059)
The word is "blur." Low-pass filtering is blurring.
Is blurring good for compression? I don't know what that means. If the image size (not the file size) is held constant, a blurry image and a clear image take up exactly the same amount space in memory.
Blurring is bad for quality. Our vision is sensitive to high-frequency stuff, and low-pass filtering is by definition the indiscriminate removal of high-frequency information. Most compression schemes are smarter about the information they filter.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by acuozzo on Friday April 26, 2024 @05:37PM
(#40174603)
> Is blurring good for compression? I don't know what that means.
Consider lossless RLE compression schemes. In this case, would data with low or high variance compress better?
Now consider RLE against sets of DCT coefficients. See where this is going?
In general, having lower variance in your data results in better compression.
> Our vision is sensitive to high-frequency stuff
Which is exactly why we pick up HF noise so well! Post-processing houses are very often presented with the challenge of choosing just the right filter chain to maximize fidelity under size constraint(s).
> low-pass filtering is by definition the indiscriminate removal of high-frequency information
It's trivial to perform edge detection and build a mask to retain the most visually-meaningful high frequency data.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by turnsout on Wednesday April 24, 2024 @12:10PM
(#40146098)
I think you're missing the point of this paper—the precise thing it's showing is upscaling previously downscaled video with minimal perceptual differences from ground truth.
So you could downscale, then compress as usual, and then upscale on playback.
It would obviously be quite attractive to be able to ship compressed 480p (or 720p etc) footage and be able to blow it up to 4K at high quality. Of course you will have higher quality if you just compress the 4K, but the file size will be an order of magnitude larger.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by turnsout on Wednesday April 24, 2024 @05:52PM
(#40150181)
In our hypothetical example, the compressed 4k data or the compressed 480p data? You would enhance the compressed 480p—that's what the example is. You would probably not enhance the 4K, because there's very little benefit to increasing resolution beyond 4K.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @11:58AM
(#40145949)
Yes. Down sampling makes only sense if you store per pixel data, which is obviously a dumb idea. You get a stream for 480p which contains frames which were compressed from the source files, or the 4k version. At some point there might have been down sampling involved, but you never actually get any of that data, you get the compressed version of those.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by p1esk on Wednesday April 24, 2024 @12:29PM
(#40146395)
Not sure if I’m being dumb, or if it’s you not explaining it clearly: if Neflix produced low resolution frames from high resolution (4k to 480p), and if these 480p frames are what my TV is receiving - are you saying it’s not downsampling, and my TV would not benefit from this new upsampling method?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @12:47PM
(#40146738)
Your TV never receives per pixel data. Why would you use a NN to enhance the data which your TV has constructed instead of enhancing the data it actually receives?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by p1esk on Wednesday April 24, 2024 @12:57PM
(#40146898)
OK, I admit I don’t know much about video compression. So what does my TV receives from Netflix if it’s not pixels? And when my TV does “upsampling” (according to the marketing) what does it do exactly?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by itishappy on Wednesday April 24, 2024 @07:22PM
(#40151155)
It receives information about the spacial frequency content of the image. If you're unfamiliar, it's definitely worth looking into the specifics of how this works, as it's quite impressive! Here's a few relevant Wikipedia articles, and a Computerphile video:
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by ALittleLight on Wednesday April 24, 2024 @12:00PM
(#40145973)
Streaming services already deliver in lower resolution than they available based on network conditions. Good upscaling would let you save on bandwidth and deliver content easier to people in poor network conditions. The tradeoff would be that details in the image wouldn't be exactly the same as the original - but, presumably, nobody would notice this so it would be fine.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by web007 on Wednesday April 24, 2024 @11:36AM
(#40145657)
Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity / with similar artifacts to the same process at 50x data volume? The world will never know.
You're also ignoring the part where all lossy codecs throw away those same details and then fake-recreate them with enough fidelity that people are satisfied. Same concept, different mechanism.
Look up what 4:2:0 means vs 4:4:4 in a video codec and tell me you still think it's "pure insanity" to rescale.
Or, you know, maybe some people have reasons for doing things that aren't the same as the narrow scope of use-cases you considered, and this would work perfectly well for them.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @11:44AM
(#40145773)
>Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity
Because you can just not downscale them and compress them in the frequency domain and encode them in 200Kbps? This is pretty obvious, seriously do you not understand what JPEG does? And why it doesn't do down sampling?
Do you seriously believe downscaling outperforms compressing in the frequency domain?
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by constantcrying on Wednesday April 24, 2024 @12:04PM
(#40146034)
Scaling color data is a different technique than down sampling. Again, all I am saying is that for a very good reason you do not stream pixel data or compress movies by storing data that was down sampled.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by web007 on Thursday April 25, 2024 @02:05AM
(#40153989)
Yes, absolutely. Paychovisual encoding can only do so much within the constraints of H.264/265.
Throwing away 3/4 (half res) or 15/16 (quarter res) of the data, encoding to X bitrate and then decoding+upscaling looks far better than encoding to the same X bitrate with full resolution.
For high bitrate, native resolution will of course look better. For low bitrate, the way H.26? algorithms work end up turning high resolution into a blocky ringing mess to compensate, vs lower resolution where you can see the content, just fuzzily.
Go get Tears of Steel raw 4K video (Y4M I think it's called). Scale it down 4x and encode it with ffmpeg HEVC veryslow at CRF 30. Figure out the bitrate, then cheat - use two-pass veryslow HEVC encoding to get the best possible quality native resolution at the same bitrate as your 4x downscaled version. You're aiming for two files that are about the same size. Somehow I couldn't convince the codec to go low enough to match, so I had the low-res version about 60% of the high-res version filesize. Now go and play them both back at 4K with just whatever your native upscale is - bilinear, bicubic, maybe NVIDIA Shield with it's AI Upscaling.
Go do that, then tell me you honestly think the blocky, streaky, illegible 4K native looks better than the "soft" quarter-res version.
Re:VideoGigaGAN: Towards Detail-Rich Video Super-Resolution
by itishappy on Wednesday April 24, 2024 @08:17PM
(#40151616)
Your video codec should never create a 480p version at all. Downsampling is incredibly lossy. Instead stream the internal state of your network directly, effectively using the network to decompress your video. Train a new network to generate this state, acting as a compressor. This is the principle of neural compression.
This has two major benefits:
1. You cut out the low resolution half your network entirely. (Go check out the architecture diagram of the original post.)
2. Your encoder network now has access to the original HD video, so it can choose to encode the high-frequency details directly instead of generating them afterwards.
I'd say most videos in practice are longer than 200 frames, so lot more research is still needed.
For animations it's around 15 seconds.
Also, it's less likely that you'd want to upscale a modern movie, which is more likely to be higher resolution already, as opposed to an older movie which was recorded on older media or encoded in a lower-resolution format.
It reminds me of the story about the Air Force making cockpits to fit the elusive average pilot, which in reality fit none of their pilots...
Another movie of him that crimes of this non stop is The Prestige.
https://en.wikipedia.org/wiki/Children_of_Men#Single-shot_se...
Funny - this is also a good description of I Am Legend.
https://www.cbr.com/i-robot-original-screenplay-isaac-asimov...
I think like the motorcycle chase that they borrowed from the lost world in Jurassic world, they also have a scene with those tiny dinosaurs pecking someone to death.
:) "Our model is proven production-ready using real-world footage from Taken 3"
https://www.youtube.com/watch?v=gCKhktcbfQM
Would it kill them to say that the method works best on short videos/scenes?
Source: https://www.nasa.gov/history/115-years-ago-wright-brothers-m....
For live streaming it's pretty common to see 2 or 3 seconds (reduces broadcast delay, but with some caveats).
0: https://dev.to/100mslive/introduction-to-low-latency-streami...
It could work for TV or movies if done properly at the scene transition time though.
Moreover I imagine that further research and power will do a lot, smarter, and quicker.
Don't forget people had toy story-comparable games in a decade or so after it was originally rendered at 1536x922.
But idk how someone can write "extremely long videos" with a straight face when meaning seconds.
Maybe "long frame sequences"
The details changing every ten seconds or so is actually a good thing; the viewer is reminded that what they are seeing is not real, yet still enjoying a high resolution video full of high frequency content that their eyes crave.
They will have huge budgets for compute and the makers of compute will be happy to absorb those budgets.
Cloud production was already growing but this will continue to accelerate it imho
the compute budgets for basic run of the mill small screen 3D rendering and 2D compositing is already massive compared to most other businesses of a similar scale. the industry has been under paying their artists for decades too.
I'm willing to bet that as soon as unreal or adobe or whoever comes out with a stable diffusion like model that can be consistent across a feature length movie, they'll stop bothering with artists altogether.
why have an entire team of actual people in the loop when the director can just tell the model what they want to see? why shy away from revisions when the model can update colour grade or edit a character model throughout the entire film without needing to re-render?
Limited access to the tech added some mystique to it too.
Just like digital cameras created a lot more average photographers, it pushed photography to a higher standard than just having access to expensive equipment.
While of course it isn't impossible for any industry to reinvent itself, movie as an art form won't die....having doubts about where it's going.
Unlikely in the next 10 years or the next 100?
With that said, I’m pretty confident that the movie industry will exist in 10 years (maybe heavily tranformed, but still existing and still pretty big). If it’s still a big part of current popculture by then (vs obviously on its way out) then I’d expect a collapse of it to require a change that is not a result of AI proliferation, but something else entirely.
Realistically, AI being able to replace Hollywood is something that could happen in 20-50 years. That's within most people's lifetime.
I don't think we're far away from models that are able to take video input of an almost finished movie and add the finishing touches.
Eg. make the lighting better, make the cgi blend in better, hide bits of set that ought to have been out of shot, etc.
The difference will be that this time the images will be crystal clear, just hallucinated by a neural network.
If you just want the picture to be pretty, this is probably cheaper than a bigger sensor.
Since this isn't lossless decompression, the point of having no "real" data is already reached. It _is_ inventing things, and the only relevant question is how plausible are the things being invented; in other words, if the video also existed in higher resolution, how close would it actually look like the inferred version. Seems obvious that this metric increases as a function of the amount of information from the source, but I would guess the exact relationship is a very open question.
The start point. Upscaling is by definition creating information where there wasn't any to begin with.
Nearest neighbor filtering is technically inventing information, it's just the dumbest possible approach. Bilinear filtering is slightly smarter. This approach tries to be smarter still by applying generative AI.
There is plenty of real information: that's what the model is trained on. That information ceases to be real the moment it is used by a model to fill in the gaps of other real information. The result of this model is a facade, not real data.
https://paperswithcode.com/task/multi-frame-super-resolution
They used digital upsampling techniques and colorization to make World War One footage into high resolution. Jackson would later do the same process for the 2021 series Get Back, upscaling 16mm footage of the Beatles taken in 1969: https://www.imdb.com/title/tt9735318/
Both of these are really impressive. They look like they were shot on high resolution film recently, instead of fifty or a hundred years ago. It appears that what Peter Jackson and his team did meticulously at great effort can now be automated.
Everyone should understand the limitations of this process. It can't magically extract details from images that aren't there. It is guessing and inventing details that don't really exist. As long as everyone understands this, it shouldn't be a problem. Like, we don't care that the cross-stitch on someone's shirt in the background doesn't match reality so long as it's not an important detail. But if you try to go Blade Runner/CSI and extract faces from reflections of background objects, you're asking for trouble.
I think it's almost inherently so, because of the care that an artist takes in choosing keyframes, deforming the action, etc.
(It doesn't hurt that a few minor hallucinations aren't going to bother anyone.)
Yes, it is impressive, but it's not what you want to actually "enhance" a movie.
This appears to happen because they begin animating as soon as they finish loading, which happens at different times for each side of the image.
https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing
The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them.
Downscaling the images and then upscaling them is pure insanity when the high res images are available.
I think it a bit unimaginative to see no use cases for this.
The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
Down sampling is a bad way to do compression. It makes no sense to do NN reconstruction on that if you could have compressed that image better and reconstructed from that data.
Are you saying low-pass filtering is bad for compression?
Is blurring good for compression? I don't know what that means. If the image size (not the file size) is held constant, a blurry image and a clear image take up exactly the same amount space in memory.
Blurring is bad for quality. Our vision is sensitive to high-frequency stuff, and low-pass filtering is by definition the indiscriminate removal of high-frequency information. Most compression schemes are smarter about the information they filter.
Consider lossless RLE compression schemes. In this case, would data with low or high variance compress better?
Now consider RLE against sets of DCT coefficients. See where this is going?
In general, having lower variance in your data results in better compression.
> Our vision is sensitive to high-frequency stuff
Which is exactly why we pick up HF noise so well! Post-processing houses are very often presented with the challenge of choosing just the right filter chain to maximize fidelity under size constraint(s).
> low-pass filtering is by definition the indiscriminate removal of high-frequency information
It's trivial to perform edge detection and build a mask to retain the most visually-meaningful high frequency data.
So you could downscale, then compress as usual, and then upscale on playback.
It would obviously be quite attractive to be able to ship compressed 480p (or 720p etc) footage and be able to blow it up to 4K at high quality. Of course you will have higher quality if you just compress the 4K, but the file size will be an order of magnitude larger.
https://en.wikipedia.org/wiki/JPEG#JPEG_codec_example
https://en.wikipedia.org/wiki/Discrete_cosine_transform
https://www.youtube.com/watch?v=Q2aEzeMDHMA
I think upscaling framerate would be more useful.
You're also ignoring the part where all lossy codecs throw away those same details and then fake-recreate them with enough fidelity that people are satisfied. Same concept, different mechanism.
Look up what 4:2:0 means vs 4:4:4 in a video codec and tell me you still think it's "pure insanity" to rescale.
Or, you know, maybe some people have reasons for doing things that aren't the same as the narrow scope of use-cases you considered, and this would work perfectly well for them.
Because you can just not downscale them and compress them in the frequency domain and encode them in 200Kbps? This is pretty obvious, seriously do you not understand what JPEG does? And why it doesn't do down sampling?
Do you seriously believe downscaling outperforms compressing in the frequency domain?
Throwing away 3/4 (half res) or 15/16 (quarter res) of the data, encoding to X bitrate and then decoding+upscaling looks far better than encoding to the same X bitrate with full resolution.
For high bitrate, native resolution will of course look better. For low bitrate, the way H.26? algorithms work end up turning high resolution into a blocky ringing mess to compensate, vs lower resolution where you can see the content, just fuzzily.
Go get Tears of Steel raw 4K video (Y4M I think it's called). Scale it down 4x and encode it with ffmpeg HEVC veryslow at CRF 30. Figure out the bitrate, then cheat - use two-pass veryslow HEVC encoding to get the best possible quality native resolution at the same bitrate as your 4x downscaled version. You're aiming for two files that are about the same size. Somehow I couldn't convince the codec to go low enough to match, so I had the low-res version about 60% of the high-res version filesize. Now go and play them both back at 4K with just whatever your native upscale is - bilinear, bicubic, maybe NVIDIA Shield with it's AI Upscaling.
Go do that, then tell me you honestly think the blocky, streaky, illegible 4K native looks better than the "soft" quarter-res version.
This has two major benefits:
1. You cut out the low resolution half your network entirely. (Go check out the architecture diagram of the original post.)
2. Your encoder network now has access to the original HD video, so it can choose to encode the high-frequency details directly instead of generating them afterwards.