News for hackers. Stuff that matters.
Re:Guess the Video Generated with AI by kungfupawnda on Friday February 23, 2024 @05:21PM (#39486905)
I was wondering on the internals of Sora
Re:Guess the Video Generated with AI by lucubratory on Friday February 23, 2024 @06:13PM (#39487360)
It's a diffusion transformer architecture, so no it doesn't piece together pieces of video. If you're familiar with denoising algorithms, the diffusion algorithm is essentially a semantically guided denoising algorithm. If you feed it pure noise so the only information it has is the semantic guiding, it will generate a video from that noise directly. I'm not sure exactly how the transformer part of the algorithm contributes, but my guess is that it's giving the denoiser the ability to not just look at adjacent pixels in 2D space, but across time through the attention mechanism. That's just a guess, though.
Re:Guess the Video Generated with AI by kungfupawnda on Friday February 23, 2024 @07:34PM (#39487935)
Thank you, after reading your comment, I did some research and stumbled upon this, an explanation of how Sora works from Jim Fan:

https://www.reddit.com/r/LocalLLaMA/comments/1aspxox/explana...