Re:Inference-Aware Fine-Tuning for Best-of-N Sampling in Lar...
by peepeepoopoo114 on Monday April 28, 2025 @02:09AM
(#43818141)
Almost all of the efficiency gains have come from shedding bit precision, but the problem is that AI labs are now running out of bits to shed. The move to reduced precision inference has been masking the insane unsustainability of compute scaling as a model improvement paradigm.
[ Parent
| on HN ]
Re:Inference-Aware Fine-Tuning for Best-of-N Sampling in Lar...
by nullc on Monday April 28, 2025 @06:02PM
(#43826617)
Is there really a limit on bits to shed? I suspect not.
Take N gates, normalize them, represent them as points on the surface of a hypersphere. Quantize the hypersphere as coarsely as you need to get the precision you want. Want less precision but your quantization is getting too coarse? Increase N.
Fast algebraic codes exist to convert positions on a hyperspheric-ish surfaces to indexes and vice versa.
Perhaps spherical VQ isn't ideal-- though I suspect it is, since groups of weights often act as rotations naturally-- but some other geometry should be good if not.