Training Text-to-Image Models Without a VAE

Re:Training Text-to-Image Models Without a VAE by joefourier on Friday October 09, 2026 @09:21PM (#50028554)

Understandable, (V)AEs are their own rabbit hole (I've spent way too much time working with them). I think they deserve more attention, there's been a few good papers recently about how to ensure that the latent space is optimal for generation without sacrificing too much reconstruction detail, but it's no easy task.

But how do you plan on scaling to 1080p24 (or higher resolution) video with end to end pixel space training? For a 10 second long video, that's 1920x more tokens than your 512x512 DiT experiments, and I'd be concerned about scaling linear patchification temporally and spatially past a certain point.

[ Parent | on HN ]

Re:Training Text-to-Image Models Without a VAE by schopra909 on Friday October 09, 2026 @11:41PM (#50029323)

It’s a good question; it’s not something that worries us too much to be honest.

The linear patchificaion is per patch, so if it works on 360p 2 seconds (really short low res) it should adapt naturally (given ROPE3D). The final layers of the DiT start moving us from latent space to pixel space.

Net net the compression is as much as H3 (32x32xTime) so it should be comparable to the best VAEs in terms of expense

[ Parent | on HN ]