Testing performance of LLM without testing the quality isn’t really practical in real world because of its fast and output is gibrish it won’t matter.

There should be 10-20 input and output that is tested for correctness or something in addition to t/s as a reference