I know it's pretty common in quant firms.

I built my own. It’s IMO necessary, but it quickly turns into a job by itself. I’ve been benching K3.1 all weekend. Even when you get a good test suite, you need to do multiple runs, not for variance, but because the model behavior changes so much between releases you often need to retune to get a real score if what you’re after is actual optimal (or closer to it) performance on your workload.

Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.

That's the way to go about it, so I invested many days of work but it is something I dread now. It was a lot of work and still feel incomplete.

Feels like the benchmark is the most important thing to be able to iterate quickly without understanding everything that happens in the code.

But needs to be taken with a grain of salt by end-user obviously