nomi_uw

I don't think it's not a problem, and I'm sure it pops up extremely often in the case of competitions when things are won by margins of +0.01x and hundreds of people are chipping at it with agents for days. This is a very different scenario to what I assume most people might be doing, though.

Thanks for bringing up these points, it's definitely good food for thought to add to the blog!

Seems like the kernel portion of that competition is closed. I can probably compare whatever the serving winner is against JamInfer when it's out (seems like they start that track on the 12th). The top of the article shows the e2e times on some math RL tasks, with fp8 weights/activations/KV, comparing against SGLang on two models. They use the same concurrent batch size, quantization scheme, and the final lengths were about the same.

30B-A3B MoE

Jam - 67,241

SGLang - 44,696

+50.4%

8B Dense

Jam - 58,821

SGLang - 41,759

+40.9%

Math RL is pure decode, so kernels are the most major factor here.

Yes, they can reward hack. However, making sure the model knows what your requirements are (e.g., "do not quantize, make sure to evict, meet this set of specifications") can mitigate that, and simply looking at the code briefly (or having the LLM describe it back to you) can mitigate it quite easily. Often, such issues are a result of insufficient specificity by the user. If you tell it just to make the model faster, it'll do it and not assume additional requirements.

Same thing applies to benchmarking, a few guardrails goes a long way. As long as you aren't completely vague ("set up benchmarks", vs. "write tests over a variety of common shapes, be careful to control for environmental variables") the models do quite well. I've checked over the code it's written and it's all fairly reasonable. I do mention cache eviction in the article, this generally wasn't an issue and the model did this automatically on the first pass (back in Opus 4.6 days). I think you might not be giving them enough credit.

I'm not completely familiar with the writers of this article, but I've never had an LLM try to hack this, even sometimes telling me a target was impossible without quantization or approximation:

"Code based analysis already tells a detailed story, but there is an overlooked parameter: "boundaries of the hardware". " - the coreauto post

This project has spanned several models since Opus 4.5. I think even then such details were trivially known by them, thought they required much more handholding than they do now. I'm honestly not sure how the article writers ran into the hacks they did.

Hmm, this is a good point. I tried to be fairly brief on this part and separated a lot of the profiling to a separate page since I usually use AI for benchmarking these days. I do think there are a lot of principles that humans should learn in practice, even if most of this is done with LLMs.

I'll update the post in the coming week with a section on benchmarking. The current post does have some details on measurement, but I agree that maybe it's not covered in the required depth.

Will update, thanks for pointing out.

I appreciate the compliment, yes this is spot on

The post is actually mostly human written! I personally am not a fan of Claude's writing and don't even use Claude (my main subscription is Codex).