Ah interesting. What's the difference? Isn't it just finding the minimal mapping of character sequences to numbers?

That's what I thought when I started working on this, but it turns out the answer is no! This approach - "minimal mapping of character sequences to numbers" - can be described as a greedy algorithm. Using this approach will often produce correct results, but not always. I can provide an example:

Input string: " grabbed"

Tokenize that with the greedy algorithm, you get [17229, 2580] == [" grab", "bed"]

Tokenize that with actual LLaMA tokenizer, you get [2646, 1327, 287] == [" gra", "bb", "ed"]

Note that the correct tokenizer represents this string with 3 tokens, even though it would be more efficient to represent this string with 2 tokens (yes, those 2 tokens exist in the vocabulary).

LLaMA uses SentencePiece Byte-Pair Encoding for tokenization, and it has many weird quirks like this.