Re:Llama from Scratch (or how to implement a paper without c...
by mike_hearn on Wednesday August 09, 2023 @12:55PM
(#37065488)
Tokens are also numbers in practice, but they're indexes into a lookup table of character sequences so yes there's very little between the two definitions. Embeddings are in turn the result of looking up that index in a table, and the result is a vector. So:
character sequence (string) -> token (small integer) -> embedding (vector of floats)
character sequence (string) -> token (small integer) -> embedding (vector of floats)