Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.

From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]

I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.

[0] https://github.com/crusoecloud/fastokens

> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache

Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.

I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.

It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.

The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.

> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.

Very cool, thanks.

Can't you tokenize in preloading on demand?

You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.

In practice every training project I've worked on does tokenization in a separate data processing phase.

If you are running on large-scale data, have you validated at that scale (comparing results)? From a quick look at the code, it looks like there is a 42-bit hash (computed via single-mul hash function) which can have collisions and thus return the wrong tokens, right?

If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.

I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)

> … only for the run to crash over some minor bug after that

Ah, Python.

An example would be setting batch size too high causing OOM which isn’t really a python problem.

Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!

That's fair, I just figure there are useful scenarios as well. Apologies if I came off as dismissive!

It didn’t come off as dismissive to me. I was curious as well as to where such optimizing helps and knew that the answers to your question would help me discover use cases I didn’t think of

It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.

It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.

But are those bounded on the speed of tokenization?

I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.

Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.

A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.

I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.