News for hackers. Stuff that matters.
Re:Binary vector embeddings are so cool by kevinventullo on Monday November 11, 2024 @12:38PM (#42108880)
It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do something like randomly flip bits with a probability weighted by the severity of the error during backprop… anyway, I’m sure there’s plenty of literature about this.
Re:Binary vector embeddings are so cool by janalsncm on Monday November 11, 2024 @02:16PM (#42109674)
Probably just an extreme version of quantization-aware training? During training you round the prediction to the range you want, but keep it as a float.

Since rounding isn’t differentiable there’s fancy techniques to approximate that as well.

> QAT backward pass typically uses straight-through estimators (STE), a mechanism to estimate the gradients flowing through non-smooth functions

https://pytorch.org/blog/quantization-aware-training/

Re:Binary vector embeddings are so cool by alok-g on Monday November 11, 2024 @02:01PM (#42109572)