I feel like we're close too, but for another reason.
For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that.
However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc.
And the scene shall be sent into Blender and you'll click on a button and have an actual rendering made by Blender, with correct lighting.
Wanna move that bicycle? Move it in the 3D scene exactly where you want.
That is coming.
And for audio it's the same: why generate an audio file when soon models shall be able to generate the various tracks, with all the instruments and whatnots, allowing to create the audio file?
That is coming too.
I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.
More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...
The only trick is that there has to be enough Blender Python code to train the LLM on.
While it generates a lot of code that initially makes sense, when you use the code, you get a jumbled block.
But also I suspect there just isn't that much openscad code in the training data, and the semantics are different enough to python or any of the other languages that are well-represented that it struggles.
That pretty much matches my experience working with NN's and LLM's
LLMs are good at writing copy that sounds accurate and creative enough, and there are known techniques to improve that (such as generating an outline first, then generating each section separately). If you then give them a list of templates, and written examples of what they are used for, the LLM is able to pick one that's a suitable match. But this is all just probability, there's no real creativity here.
Earlier this year I played around with trying to have GPT-3 directly output an SVG given a prompt for a simple design task (a poster for a school sports day), and the results were pretty bad. It was able to generate a syntantically coreect SVG, but the design was terrible. Think using #F00 and #0F0 as colours, placing elements outside the screen boundaries, layering elements so they are overlapping.
This was before GPT-4, so it would be interesting to repeat that now. Given the success people are having with GPT-4V, I feel that it could just be a matter of needing to train a model to do this specific task.
Almost all of the generative 3D models you see are actually generative image models that essentially (very crude simplification) perform something like photogrammetry to generate a 3D model - 'does this 3D object, rendered from 25 different views, match the text prompt as evaluated by this model trained on text-image pairs'?
This is a shitty way to generate 3D models, and it's why they almost all look kind of malformed.
You wouldn't need any models to learn from. But my intuition is that RL is still quite weak, and that the model would flounder after learning to mimic background color and placing a few spheres.
Probably because they aren't as advanced and the demos aren't as impressive to nontechnical audiences who don't understand the implications: there’s lots of work on text-to-3d-model generation, and even plugins for some stable diffusion UIs (e.g., MotionDiff for ComyUI.)
For single 3D object the biggest dataset is ObjaverseXL with 10M samples
For full 3D scenes you could at best get ~1000 scenes with datasets like ScanNet I guess
Text2Image models are trained on datasets with 5 billion samples
Got a public dataset?
I've not worked in VFX for a while, but when I did the modeling departments at multiple studios had giant libraries of completed geometries for every project they ever did, plus even larger libraries of all the pieces and parts they use as generic lego geometry whenever they need something new.
Every 3D modeler I know has their own personal libraries of things they'd made as well as their own "lego sets" of pieces and parts and generative geometry tools they use when making new things.
Now this is just a guess, but do you know anyone going through one of those video game schools? I wager the schools have big model libraries for the students as well. Hell, I bet Ringling and Sheridan (the two Harvards of Animation) have colossally sized model libraries for use by their students. Contact them.
So we could have something to convert AI-generated image output into 3D scenes without having to explicitly train the "creative" AI for that.
Probably much more viable, because the quantity of 3D models out in the wild is far far lower than that of bitmap images.
I agree with this philosophy - Teach the AI to work with the same tools the human does. We already have a lot of human experts to refer to. Training material is everywhere.
There isn't a "text-to-video" expert we can query to help us refine the capabilities around SD. It's a one-shot, Jupiter-scale model with incomprehensible inertia. Contrast this with an expert-tuned model (i.e. natural language instructions) that can be nuanced precisely and to the the point of imperceptibility with a single sentence.
The other cool thing about the "use existing tools" path is that if the AI fails part way through, it's actually possible for a human operator to step in and attempt recovery.
It can learn anything you have data for.
Heck, we do it with geospatial data already, generating segmentation vectors. Why not 3D?
Not in theory, but the level of complexity is way higher and the amount of data available is much smaller.
Compare bitmaps to this: https://fossies.org/linux/blender/doc/blender_file_format/my...
But we haven't gotten diffusion working well for text/code, so generating long files is a problem.
I'm not experienced enough to validate their claims, but I love the choice of languages to evaluate on:
> Python, Bash and Excel conditional formatting rules.
A 3D scene is vastly more complex, and the way you consume it is tangential to the rendering of it we use to interpret. It is a collection of arbitrary data structures.
We’ll need a new approach for this kind of problem
> A 3D scene is vastly more complex
3D scenes, in fact, are also data, numbers and tokens. (Well, numbers, but so are tokens.)
Not at all the same as fixed sized arrays representing images.
In fact, the data structures of a 3D scene can be serialized as text, and a properly trained text gen system could generate such a representation directly, though that's probably not the best route to decent text-to-3d.
Serializing 3D models as text is not going to work for negligibly non trivial circumstances.
The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format.
You need a whole metric fuckton of these.
After that, you train your model to produce those 3d scene files instead of image bitmaps.
You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data.
I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed.
Sorry, I'm just rambling here but its very exciting stuff.
The question is whether the 99% of the audience would even care...
As for "the internet", there will always some small part of it which will obsess and/or laught over anything, doesn't mean they represent anything significant - not even when they're vocal.
Perhaps a more computationally expensive but better looking method will be to pull all objects in the scene from a 3D model library, then programmatically set the scene and render it.
However, 3D is just one approach to rendering visuals. There are so many other styles and methods how people create images, and if I understand correctly, we can do image-to-text to analyze image content, as well as text-to-image to generate it - regardless of the orginal method (3d render or paintbrush or camera lens). There are some "fuzzy primitives" in the layers there that translate to the visual elements.
I'm hoping we see "editors" that let us manipulate / edit / iterate over generated images in terms of those.
In the end diffusion technology can make a more realistic image faster than a rendering engine can.
I feel pretty strongly that this pipeline will be the foundation for most of the next decade of graphics and making things by hand in 3D will become extremely niche because lets face it anyone who has worked in 3D it's tedious, it's time consuming, takes large teams and it's not even well paid.
The future is just tools that give us better controls and every frame will be coming from latent space not simulated photons.
I say this as someone who had done 3D professionally in the past.
Diffusion models aren't LLMs (they may use something similar as their text encoder layer) and they simulate their training corpus, which usually isn't selected solely for physical fidelity, because that's not actually the single criteria for visual imagery outside of what is created by diffusion models.
Do the parameters think that Jazz musicians are mormon? Padres often surf? Wizards like the Lincoln Memorial?
Emu can do that.
The bluejay/toronto thing may be addressable later (I suspect via more detailed annotations a la dalle3) - these current video models are highly focused on figuring out temporal coherence
Also, maybe you can't edit post facto, but when you give prompts, would you not be able to say : two blue jays but no CN tower
https://emu-edit.metademolab.com/
https://ai.meta.com/blog/emu-text-to-video-generation-image-...
This is not the flex you think it is. You don't have to like sports, but snarking on people who do doesn't make you intellectual, it just makes you come across as a douchebag, no different than a sports fan making fun of "D&D nerds" or something.
Rather than projecting your own hangups and calling people names, try instead assuming that they're not trying to offend you personally and are just using common vernacular.
Also, I originally tried to get the 3 characters in the image to be generated simultaneously, but eventually gave up as DALL-E had a hard time understanding how I wanted them positioned relative to each other. I just generated 3 separate characters and positioned them in the same image using Gimp.
Nearly all of the available models have this, even the highly commercialized ones like in Adobe Firefly and Canva, it’s called inpainting in most tools.
See video: https://www.adobe.com/max/2023/sessions/project-stardust-gs6...
Yeah. They're not "videos" so much as images that move around a bit.
This doesn't really look any better than those Midjourney + RunwayML videos we had half a year ago.
>Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close.
Google has a model called Phenaki that supposedly allows for that kind of stuff. But the public can't use it so it's hard to say how good it actually is.
As for your last question yes that exists. There are two models from Meta that do exactly this, instruction based iteration on photos, Emu Edit[0], and videos, Emu Video[1].
There's also LLaVa-interactive[2] for photos where you can even chat with the model about the current image.
[0]: https://emu-edit.metademolab.com/
I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue jays" would be near "toronto" or "cn tower". The improvements in scale and speed (image -> now video) are impressive, but given how incredibly able the image generation models are, they simultaneously feel crippled and limited by their lack of editing / iteration ability.
Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close.