I would think it is related to the fact that Stable Diffusion can run on consumer level hardware, whereas the largest language models don't, as they need hundreds of Gigs of GPU memory.