News for hackers. Stuff that matters.
Re:Llama from Scratch (or how to implement a paper without c... by mike_hearn on Wednesday August 09, 2023 @12:55PM (#37065488)
Tokens are also numbers in practice, but they're indexes into a lookup table of character sequences so yes there's very little between the two definitions. Embeddings are in turn the result of looking up that index in a table, and the result is a vector. So:

character sequence (string) -> token (small integer) -> embedding (vector of floats)

There are no comments on this comment yet.