Re:Can GPT-4 and GPT-3.5 play Wordle?
by pallas_athena on Tuesday March 21, 2023 @03:33AM
(#35243428)
An Embedding is a n-dimensional vector (think of it as a sequence of n numbers).
During training, each token (or word) gets an Embedding assigned.
Critically, _similar words will get similar embeddings_. And "similar words" could mean both semantically or (as was the example) syntactically ("apple" and "appli").
And being vectors, you can do operations on them. To give the classic example, you could do:
Embedding(`king`) + Embedding(`female`) = Embedding(`queen`).
During training, each token (or word) gets an Embedding assigned.
Critically, _similar words will get similar embeddings_. And "similar words" could mean both semantically or (as was the example) syntactically ("apple" and "appli").
And being vectors, you can do operations on them. To give the classic example, you could do: Embedding(`king`) + Embedding(`female`) = Embedding(`queen`).