Would a possible solution be a hybrid approach? I remember back years ago seeing AI research around predicate logic, with hundreds of thousands of classification entries and "x can't be y" type stuff.

Maybe the potential output of an LLM could be run through that kind of logic and fed back to itself for sanity before being decided as final output?