A primer on GPUs and kernel optimization

Re:A primer on GPUs and kernel optimization by Farmadupe on Monday November 09, 2026 @07:38PM (#50434541)

As someone who loves being able to press a button to receive matmuls tuned to the 3090 by his feet, are there rules of thumb to work out how to set thresholds for evals?

Do I want to set a particular threshold on any individual op (or a layer for a given transformer)? e.g "for this test set, 99% of l2norms being within 1e-4 error of the golden reference kernel"

Or do I want to just shortcut all that and go with task based evals? eg "word error no worse than 1% greater than the reference speech-to-text model"?

I'm assuming the latter is the way to go in the first instance?

[ Parent | on HN ]

Re:A primer on GPUs and kernel optimization by augment_me on Monday November 09, 2026 @08:00PM (#50434705)

It depends on if you do training or inference and also at what scale.

For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.

Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.

This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.

Numerics and correctness are more important for training kernels and backwards passes

[ Parent | on HN ]

Re:A primer on GPUs and kernel optimization by Farmadupe on Monday November 09, 2026 @08:20PM (#50434846)

Thanks! Interestingly I find that coding agents really don't now how to quantize a model to a given loose target. "Make it faster and accurate" is seemingly in distribution, "Make it _even_ faster and worse" is a good way to get Opus 5.5 to say "I tried nothing and I'm all out of ideas"

[ Parent | on HN ]