I’ve had similar thoughts, wondering why LLMs aren’t just a component of a larger system, perhaps used to generate “thoughts” that are then tested by some other subsystems, which in turn may re-fire the LLMs to generate more.

I guess I’m imagining that testing the LLM output might be a problem that some other technique might be suited for, and smells like what we need to mitigate LLM weaknesses.