People can generate a few good samples of what they want a character to look like and then interrogate clip to get a more detailed prompt that makes it more consistent. It increases the prompt sizes a lot, but I don't think there's an easy way to solve this.

Maybe you could build a UI that semi-automates this process?