News for hackers. Stuff that matters.
Re:Llama from Scratch (or how to implement a paper without c... by sundarurfriend on Wednesday August 09, 2023 @12:02PM (#37064699)
Another learner here, one clarification that I think is useful even for beginners:

> A token is a unique integer identifier for a piece of text.

A token is a word fragment that's common enough to be useful on its own - for eg., "writing", "written", "writer" all have "writ", so "writ" would be an individual token, and "writer" might be tokenized as "writ" and "er".

An embedding is where the tokens get turned into unique numeric identifiers.

Re:Llama from Scratch (or how to implement a paper without c... by mike_hearn on Wednesday August 09, 2023 @12:55PM (#37065488)
Tokens are also numbers in practice, but they're indexes into a lookup table of character sequences so yes there's very little between the two definitions. Embeddings are in turn the result of looking up that index in a table, and the result is a vector. So:

character sequence (string) -> token (small integer) -> embedding (vector of floats)

Re:Llama from Scratch (or how to implement a paper without c... by sva_ on Wednesday August 09, 2023 @03:34PM (#37067398)
The tokens are in this case actually the individual characters:

    vocab = sorted(list(set(lines)))