News for hackers. Stuff that matters.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by londons_explore on Thursday June 08, 2023 @03:33PM (#36247468)
I think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance.

Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...

[1]: https://help.openai.com/en/articles/6824809-embeddings-frequ...

Re:Weird GPT-4 behavior for the specific string “ davidjl” by jxmorris12 on Friday June 09, 2023 @09:26AM (#36257505)
This comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)