Your validation approach doesn't really change based on the classification method (LLM vs NLP).

At that volume you're going to use automated tests with known correct answers + random sampling for human validation.