Re:Weird GPT-4 behavior for the specific string “ davidjl”
by jxmorris12 on Friday June 09, 2023 @09:26AM
(#36257505)
This comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)
Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...
[1]: https://help.openai.com/en/articles/6824809-embeddings-frequ...