News for hackers. Stuff that matters.

A comment on LLaMA: A foundational, 65B-parameter large language model.

Re:LLaMA: A foundational, 65B-parameter large language model by dentalperson on Friday February 24, 2023 @03:11PM (#34929450)
How does token->embedding lookup work with 1.4T BPE tokens? Since there are more tokens than the 65B parameters it must be doing some sort of interesting thing based on the merge operations. Is it different from what other GPT models with ~100k tokens are doing?

At inference, how many of those tokens are used? (they mention most tokens are used only once during training, so the must be very long sequences.)

[ Parent | on HN ]
Re:LLaMA: A foundational, 65B-parameter large language model by bitRAKE on Friday February 24, 2023 @03:47PM (#34929907)
The 1.4T tokens are what the model was trained on, and not the token range of the embedding.
[ Parent | on HN ]
Re:LLaMA: A foundational, 65B-parameter large language model by dentalperson on Friday February 24, 2023 @04:17PM (#34930264)
Ah, that makes more sense, thank you. Since this was mentioned in the tokenizer section and the number of unique tokens wasn't mentioned I misunderstood.
[ Parent | on HN ]