News for hackers. Stuff that matters.
Re:AI Content Generation, Part 1: Machine Learning Basics by tomComb on Monday September 12, 2022 @09:47PM (#32819842)
“If you’ve been following the red-hot AI content generation scene, then you’re probably already aware that it’s likely a matter of months before authentic-looking video of newly invented Seinfeld bits like this can be generated by anyone with a few basic, not-especially-technical skills.”

A matter of months eh?

To me that claim looks to be about driving up business. I realize lots of people and companies are making similar claims, and I have no doubt we will get there in a reasonable amount if time, but I believe that people with your level of expertise believe it will be at least at year until we achieve what you decide.

This whole thing is remarkable but now people are overselling it IMO.

Re:AI Content Generation, Part 1: Machine Learning Basics by nodja on Monday September 12, 2022 @11:09PM (#32820267)
Depending on how you're measuring it, video generation will come in less than 12 months, or it's already here.

- In the Video Diffusion paper[1] they generate 64x64 16 frame videos, model is not released, but an open source implementation using an imagen-like pipeline exists[2], no model is available publicly.

- The CogVideo paper[3], which is essentially a huge transformer model, can generate 480x480 videos, the code for the paper and models are open source[4], but be warned that you need a huge GPU like an A100 to run the damned thing.

The future of text to video generation will probably be video diffusion, i.e. using 3D UNets, or more likely a much more optimized version of a UNet. Improvements on diffusion models are happening on a daily basis, and we're probably a couple papers away from having a breakthrough in efficiency that we can generate good looking videos on a high end GPU.

[1] https://arxiv.org/abs/2204.03458 [2] https://github.com/lucidrains/imagen-pytorch/tree/main/image... [3] https://arxiv.org/abs/2205.15868 [4] https://github.com/THUDM/CogVideo

Re:AI Content Generation, Part 1: Machine Learning Basics by nl on Monday September 12, 2022 @11:21PM (#32820325)
It's very likely that realistic looking (to the level of Stable Diffusion) video will happen and tools to create it will be available within 12 months (maybe 80% likelihood).

What is likely to be missing is the ability to control that video in useful ways directly from prompts. There will be some type of direction, but as anyone who has spent time doing prompt-based image generation actual control isn't there yet.

Re:AI Content Generation, Part 1: Machine Learning Basics by in3d on Tuesday September 13, 2022 @12:59AM (#32820812)
I’ll take the 20%. Text-to-video won’t be good enough to be called realistic in a year.
Re:AI Content Generation, Part 1: Machine Learning Basics by nl on Friday September 23, 2022 @08:18AM (#32950904)
I admit this gave me pause when you said this. But I'd note my comment was pretty specific:

> It's very likely that realistic looking (to the level of Stable Diffusion) video will happen and tools to create it will be available within 12 months (maybe 80% likelihood).

> What is likely to be missing is the ability to control that video in useful ways directly from prompts.

ie, I'm not claiming it will entirely text based.

As an example of the kind of things we'll see I'd point at https://twitter.com/karenxcheng/status/1564626773001719813

This is using live video as a source, but I think an integrated version of this combined with some kind maybe game based interface to script it is achievable.

Re:AI Content Generation, Part 1: Machine Learning Basics by nodja on Tuesday September 13, 2022 @12:52AM (#32820765)
> but as anyone who has spent time doing prompt-based image generation actual control isn't there yet

It's not, but there are many efforts to fix this and they solve most problems, although I admit they're not ideal. One way you can control generation is simply by messing around with per token weights, for example if your prompt is being partially ignored, you can use weights to have the guidance give more emphasis on that part of the prompt. Another way you can control it is by using img2img and input as simple drawing, this can help it better understand say, which color you want things to be. The best tool of all is of course, copious amounts of inpainting and doing composition, eyes not the right color? inpaint them in, messed up hand? inpaint, etc. If your issue is that you can't generate a specific character or style at all, you have tools like text-inversion that can create a special token that represents the rough idea, assuming you already have a couple images that represent that idea already.

There is a real fix tho, the big issue with these prompts is that they use the clip text encoder, the clip text encoder was trained only on image captions, which means that it's understanding of the world is limited to whatever is represented by captions that exist on images found on the internet, a very limited subset of language, limiting the quality of the generated embeddings as the model not only is bad at basic language, it doesn't properly understand the relationships between words and such. Image models that use a large language model (LLM) listen to the prompt much better, since these LLM models generate much less noisy embeddings, a LLM contains embeddings big enough that the diffusion model can actually spell, i.e. if you ask for a sign that says something, the sign will come out with proper spelling and font choice. Sadly there's no such model open to the public yet, but stability.ai is currently training such a model and we can expect its release hopefully before christmas.

Having a image model trained with a LLM text encoder and then applying the tricks used to improve current models will give you an unprecedented level of control over image generation, the next step is probably having an instruction based model that takes both an image and text as input, and runs the instruction over the image. Example, you give it a image of a person and you send the instruction of "draw this in the style of pixel art" and it'll do it for you, it's already currently possible with img2img, but having a model than listens to instructions can extend this to more useful things like "remove the background", "add another cat", "tilt the sign 90 degrees", "make everything black and white except her dress", etc.

Re:AI Content Generation, Part 1: Machine Learning Basics by holoduke on Tuesday September 13, 2022 @11:30AM (#32825590)
Could you for example create a simple 3d render with a very simple environment with cubes resembling houses and blobs as trees etc and a camera moving through that environment. Then use video generation in way like img2img works? That would be very great.
Re:AI Content Generation, Part 1: Machine Learning Basics by nodja on Tuesday September 13, 2022 @08:31PM (#32831858)
Yup, that would be easily doable, the code would essentially be the same.
Re:AI Content Generation, Part 1: Machine Learning Basics by amrrs on Tuesday September 13, 2022 @12:34AM (#32820670)
Stable Diffusion Video generation is kind of here[1] I guess Stability Team will partner up with more open source devs to accelerate this.

[1] https://www.youtube.com/watch?v=yyxgv6MxSDk

Re:AI Content Generation, Part 1: Machine Learning Basics by nodja on Tuesday September 13, 2022 @12:58AM (#32820809)
This is not really what is being suggested, this repo just navigates the latent space between 2 prompts and generates an image at specific intervals. It create cool effects but will never be able to generate a coherent video of for example "a man walking on a beach".
Re:AI Content Generation, Part 1: Machine Learning Basics by thomasahle on Tuesday September 13, 2022 @01:34AM (#32820964)
>> it’s likely a matter of months

> A matter of months eh?

Right. As always people overestimate the effect of a technology in the short run and underestimate the effect in the long run.

It was 15 months between Dall-E and Dall-E 2. And it's already been 5 months since the release of Dall-E 2.

To get homemade Seinfeld bits we'll presumably have to wait for some big research org to release a paper and then it's a matter of months on top of that.

Unless hobbyists can take the current diffusion technology from images and simple videos to complicated videos and video + sound. But I doubt that.

Re:AI Content Generation, Part 1: Machine Learning Basics by mudrockbestgirl on Tuesday September 13, 2022 @02:25AM (#32821203)
> but I believe that people with your level of expertise

What expertise? It's just a marketing article written by a journalist who writes about all kinds of stuff. It's not written by someone with actual expertise in building models or doing research in the field. I'm surprised this marketing piece is even being upvoted because it's void of anything technical or deep.

Re:AI Content Generation, Part 1: Machine Learning Basics by rco8786 on Tuesday September 13, 2022 @07:39AM (#32823004)
This seems a bit pedantic? “A matter of months!? Psh, more like a year”

Either is insanely fast and the distinction is not particularly notable.

Also a matter of months seems an entirely reasonable estimate given the insane velocity around this stuff. I’ve already seen some video generation models.

Re:AI Content Generation, Part 1: Machine Learning Basics by tomComb on Tuesday September 13, 2022 @10:09AM (#32824490)
I'll be pedantic again ... you changed what I wrote from "at least" a year to "more like" a year, which is actually quite different.
Re:AI Content Generation, Part 1: Machine Learning Basics by godelski on Monday September 12, 2022 @11:10PM (#32820274)
So we can already make video. But the "authentic-looking" part is going to be quite a bit. Even current face work still has a lot of issues that are a bit more subtle. The diversity in a whole scene is higher. Then also you'd need convincing voice deep fakes, which does exist. But then you need to integrate this and/or the lip syncing into the prompts (totally doable). Right now peopel are trying to smooth variance between frames.

I'd suspect we'll get something 360p-480p quality in <2 years (definitely less than 5). Idk, is 24 months "a matter of months"?

Re:AI Content Generation, Part 1: Machine Learning Basics by mholm on Monday September 12, 2022 @10:05PM (#32819960)
I’m of similar opinion. I keep seeing claims of smooth AI video at the level of existing image generation ‘coming soon’ but I haven’t seen any evidence that it’s even close. Few years? Sure.
Re:AI Content Generation, Part 1: Machine Learning Basics by mkaic on Monday September 12, 2022 @10:31PM (#32820093)
I expect short video generation to be at the level of DALLE 2 in 1 year — e.g. able to get the gist of a prompt, but with lots of artifacts, requiring a lot of compute, and frequently ignoring large parts of the prompt.
Re:AI Content Generation, Part 1: Machine Learning Basics by doopy1 on Monday September 12, 2022 @10:14PM (#32820011)
I think we can have it very quickly if someone threw a ton of money at it. The question is who wants to take that bet and run at a loss for ages.
Re:AI Content Generation, Part 1: Machine Learning Basics by seibelj on Monday September 12, 2022 @10:41PM (#32820133)
Self driving cars followed the same pattern - the demo is decent but the 99.9999% case necessary for real-world usage is perpetually out of reach
Re:AI Content Generation, Part 1: Machine Learning Basics by metadat on Monday September 12, 2022 @10:46PM (#32820158)
What is acceptable for image and content generation has almost no relation to what is acceptable for safety-critical systems.
Re:AI Content Generation, Part 1: Machine Learning Basics by seibelj on Monday September 12, 2022 @11:12PM (#32820286)
Given how insanely high the bar is in creative industries, I’m not sure how deep a purely AI generated graphic can succeed
Re:AI Content Generation, Part 1: Machine Learning Basics by ChadNauseam on Tuesday September 13, 2022 @08:14PM (#32831760)
I don’t think the bar is uniformly that high. Go look at the covers of some Kindle Unlimited titles – Midjourney art is better