Re:A not so fast implementation of cosine similarity in C++ ...
by hansvm on Wednesday October 16, 2024 @10:03AM
(#41859152)
Python's floats are usually doubles by default, so it's mostly fine.
That said, yeah, that implementation isn't ideal. At a minimum, Kahan summation is usually free on large vectors (you're bottlenecked on memory bandwidth anyway), give or take the fact that you need to disable floating point re-ordering to keep the compiler from screwing it up and therefore have to order the operations correctly to make it efficient (see some other top-level comments about data dependencies as an example).
Isn't it pretty bad for accuracy to accumulate large numbers of floats in this fashion? o.O In the example it's 640,000 numbers. log2(640,000) is ~19.3 but the significand of a float has only 23 bits plus an implicit one.