Re:Stable Video Diffusion
by dragonwriter on Tuesday November 21, 2023 @05:53PM
(#38371688)
> Text, audio, and bitmapped images are data. Numbers and tokens.
> A 3D scene is vastly more complex
3D scenes, in fact, are also data, numbers and tokens. (Well, numbers, but so are tokens.)
As I stated and you selectively omitted, 3D scenes are collections of many arbitrary data structures.
Not at all the same as fixed sized arrays representing images.
Re:Stable Video Diffusion
by dragonwriter on Tuesday November 21, 2023 @07:38PM
(#38372863)
Text gen, one of the things you contrast 3d to, similarly isn't fixed size (capped in most models, but not fixed.)
In fact, the data structures of a 3D scene can be serialized as text, and a properly trained text gen system could generate such a representation directly, though that's probably not the best route to decent text-to-3d.
Text is a standard sized embedding vector that gets passed one at a time to an LLM. All tokens have the same shape. Each token is processed one at a time. All tokens also have a pre defined order. It is very different and vastly simpler.
Serializing 3D models as text is not going to work for negligibly non trivial circumstances.
A 3D scene is vastly more complex, and the way you consume it is tangential to the rendering of it we use to interpret. It is a collection of arbitrary data structures.
We’ll need a new approach for this kind of problem