What it would need to be able to do to get from here to there is understand some concepts. The first being "characters". On reddit there was beautiful image that recently won first place in an art contest and its quite frustrated some of the art community. When I was looking at it I thought it was awesome, but wondered at the ability to create another hundred or so images in that same 'world' that the created image was showing. I would want to do something like give it the prompt "tired old medieval knight with a mace and shield" and have it create the character then be able to name it "Tom" or something and feed it more prompts for that characters like "Tom is sitting in a forest brooding" and have it create the same exact character but in a different context.
That would be pretty game changing for opening up amature web comics to a large body of people who have ideas and tell stories but have no art skills to speak of - my stick characters are crooked :(
Last week PhilFTW explained "How To Create a Complete Graphic Novel in ONE Day" with Midjourney in a YouTube video [1]. He uses five tools:
- Midjourney (to generate images)
- InferKit (to generate the story text)
- Word (to rearrange the story text to fit into some narrative)
- Comic Life 3 for iPad (to place the images and text in comic book panels)
- Affinity Designer (to design the cover and export everything to print, Kindle, and Blurb)
Another trick to approach this problems is specifying the random seed, this will cause the same image being generated by the same prompt without any randomness. When you now change the prompt you get an image that is very similar to the first one, but with the variation included. Somebody used that to age a woman across 100 years[2] with quite stunning results. Even works with gender or style changes.
[1] https://textual-inversion.github.io/
[2] https://www.reddit.com/r/StableDiffusion/comments/wq6t5z/por...
[3] https://www.reddit.com/r/StableDiffusion/comments/wq6t5z/por...
It will take a lot of compute to compile the script and render the video.
It's only a few frames, but they are entirely generated from text - no seed image or interpolation required.
I'm sure these types of capabilities will come at some point, but no current model can do it. It requires more than just projecting motion into a scene.
You also could put a language model on top of your prompting system. So "gandolff kicking ass" gets translated into " Page XXX, Paragraph XX from LOTR "
This will be a huge game changer when it occurs. Whether it be for deep fake videos, creating custom content, or making a new season of your favorite tv show that was cancelled too early. The possibilities are endless.
This is probably not in the near future (i.e. this year), but I doubt it is very far off.