News for hackers. Stuff that matters.
Re:Language Models Are Injective and Hence Invertible by zipy124 on Thursday October 30, 2025 @11:28AM (#45761115)
for 768 dimensions, you'd still expect to hit (1-1/N) with a few billion samples though. Like that's a 1/N of 0.13%, which quite frankly isn't that rare at all?

Of course are vectors are not only points in one coordinate axes, but it still isn't that small compared to billions of samples.

Re:Language Models Are Injective and Hence Invertible by nyeah on Thursday October 30, 2025 @11:45AM (#45761321)
2^768 is much, much larger than 10^9. On the other hand there's the "birthday matching" thing. It's not 100% transparent to me which way this goes. Have you stepped through the orders of magnitude involved?
Re:Language Models Are Injective and Hence Invertible by sigmoid10 on Wednesday November 05, 2025 @08:16AM (#45822506)
Bear in mind that these are not base vectors at this stage (which would indeed give you 1/768). They are arbitrary linear combinations. There are exponentially many near orthogonal of these vectors for small epsilon. And epsilon is chosen pretty small in the paper.