There exists software to reproject texture normals back on to a high poly model. So this problem does have a solution for anyone interested.

That's assuming your generator produces a normal map, the ones I've seen do not, the only texture channel they output is color. That being the one channel that a model trained on images is naturally equipped to produce.

I may be speaking out of ignorance here, but couldn't you use photogrammetry techniques to translate these to a higher resolution mesh?

Only if you have multiple images of the same areas so that you can extract actual position. And there is no guarantee that multiple pictures of the same model have the same detail, much less in a manner that can be triangulated with accuracy. A lot of the photogrammetry algorithms discard points that don't match certain error-bars.

So yes, there might be a wooden frame in the middle of that window, but does it match the math on both angles of it? Doubt it.

You can generate pretty reliable texture depth maps from just an image. It’s going to be trash if you’re trying to generate the depth for the entire 3D model but I presume it’s going to go a good job with just texture. Then you just use a displacement based on the depth map.

Therefore, what is the main usecase of this model? Generating cheap 3D assets for videogames?

I don't think they have a specific use-case for this model, they're throwing ideas at the wall again in the hopes some of them stick and eventually turn into another product. The paper doesn't discuss any of the problems that would need to be solved in order to easily generate game-ready assets so I think it's safe to assume that it currently doesn't.

For games at the very least you need to consider polygon budget, getting reasonably good UVs, and generating materials which fit into a PBR shader pipeline, at least if it's going to work with rendering pipelines as we know them today (as opposed to rendering neural representations directly, which is a thing people are trying to do but is totally unproven in production).

I'd be willing to bet you could create a diffusion model to map unrefined meshes to UV-fixed and remeshed surfaces. If you had a large enough library of good meshes you just programmatically mess 'em up and use that as the dataset.

It’s always been this. None of these ever show the untextured model.

When I see a demo where they are showing wireframes I know it’ll be good enough.

Seems like a tougher nut to crack than image generation was, since there isn't a bajillion high quality 3D models lying around on the internet to use as training data, everyone is trying to do 3D model generation as a second-order system using images as the training data again. The things that make 3D assets good, the tiny geometric details that are hard to infer without many input views of the same object, the quality of the mesh topology and UV mapping, rigging and skinning for animation, reducing materials down to PBR channels that can be fed into a renderer and so on aren't represented in the input training data, so the model is expected to make far more logical leaps than image generators do.

It almost seems easier, in that you have an arbitrary # of real world objects to scan and the hardware is heavily commoditized (IIRC iPhones have this built in at highres now?)

How is building a dataset easier than using a prebuilt dataset?

In context, the conversation was beyond a dichotomy - thankfully. Having only 2 choices leaves conversation at people insisting one is better, and becomes an argument about definitions where people take turns alternating being "right" from the viewpoint of a neutral observer.

It's proposing a solution to the author's observation that everyone is doing it in second order fashion and missing a significant amount of necessary data.

The implication is that rather than doing it the hard way via the already-obtained 2nd order dataset, it'll be easier to get a new dataset, and getting that dataset will be significantly easier that it was to get the second-order dataset, as you don't need to worry about aesthetic variety as much as teaching what level of detail is needed in the mesh for it to be "real"

> since there isn't a bajillion high quality 3D models lying around on the internet to use as training data

There aren't a bajillion high-quality 3D models of everything, but there are an unbounded number of high-quality 3D models of some things, due to the existence of procedural mesh systems for things like foliage.

You could, at the very least, train an ML model to translate images of jungles into 3D meshes of the trees composing them right now.

Although I wonder if having a few very-well-understood object types like these, to serve as a base, would be enough to allow such a model to deduce more generalized rules of optics, such that it could then be trained on other object categories with much smaller training sets...

I know where I could get several hundred terabytes (maybe an exabyte? It’s constantly growing) of ultra high quality STL files designed for 3D printing. I just don’t have the storage or the knowledge of how to turn those into a model that outputs new STL files.

I’d imagine it’d require a ton of tagging, although I have a good idea of how I could leverage existing APIs to tag it mostly automatically by generating three still image thumbnails of the content, then feeding that through CLIP, and verifying that all two or three agree on what it’s an STL of, and manually tag the ones that fail that test.

There’s a pretty big difference between hundreds of terabytes and an exabyte. Maybe you meant petabyte?

Couldn’t a deep network learn the latent 3D representation just on video input?

Yes. But it is still promising. Things are getting incrementally better.

(I dream of the day when this can be used to automatically create paper-craft templates.)

There are AI models who can create proper meshes though.

Which ones?

This for example: https://research.nvidia.com/labs/toronto-ai/flexicubes/

Thanks, that's really interesting!