For what is worth, people were trying the same thing with GANs (I also played with doing it with stylegan a bit) but the results weren't as good.

The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.