On the other hand, diffusion models seem to have replaced GANs for image synthesis, so perhaps there's something I'm missing, or perhaps there's a way to combine both techniques.
Although an adversarial extra-finger detector seems in order! Extra fingers are not creativity. They are taboo!!
But for text, there is a lot of structure around bullshitting, and fortunately, the internet is full of examples of people calling bullshit. As long as the bullshit adversary has to give a strong critique to back up any bullshit call, it should work.
And bullshit judgments can be de-bullshitted too.
This would be the same as what our social circles, work colleagues, families, etc. do for us, so a circular firing squad of bullshit judges, who also give gold stars for quality responses, seems like a natural solution for models too.
I'm not worried about using approaches like this to converge on truth (model A generates an answer to a question, model B attempts to refute model A). Perhaps we could train model A to generate lies that model B can't detect.
Post train one LLM to please another LLM, that rates the quality of the first model's responses and calls it on any bullshit! (And vice versa.)