News for hackers. Stuff that matters.
Re:Kimi K3 is now live by leecommamichael on Thursday July 16, 2026 @11:28AM (#48935892)
Tokenizers define the alphabet on which the language model is trained. I don't want people to get the impression it's a module which can be swapped out or modified on its own. Alphabet size is a design consideration related to correctly encoding the training data.
Re:Kimi K3 is now live by smallerize on Thursday July 16, 2026 @12:03PM (#48936378)
That's true, but it makes it difficult to compare pricing when it's based on tokens. Maybe we need a benchmark for price per a specific input, like enwiki8.
Re:Kimi K3 is now live by leecommamichael on Thursday July 16, 2026 @12:23PM (#48936634)
Yes, almost all work people share which seeks to measure the capabilities and differences of models needs to get more precise. We are clamoring to say something meaningful about these things.
Re:Kimi K3 is now live by victorbjorklund on Thursday July 16, 2026 @01:05PM (#48937194)
It is kind of a shame we ended up comparing token pricing across models and providers when it doesn’t really make sense. Not sure what would be better though.
Re:Kimi K3 is now live by alain94040 on Thursday July 16, 2026 @01:11PM (#48937280)
Use price per page (standard English text)? That would also help make the metric easier to visualize.

If you think a page is too vague, use a famous known writer's work as a reference.

Re:Kimi K3 is now live by whoopdeepoo on Thursday July 16, 2026 @01:11PM (#48937283)
Well isn't that what benchmarks are for? They compare total cost for a unit of work.
Re:Kimi K3 is now live by whodatbo1 on Thursday July 16, 2026 @01:53PM (#48937870)
A better metric is price per byte. Most thinking traces, prompts, skills are in plain English, which is roughly 1 byte per character, assuming UTF-8 encoding (even code should not be much more either). As an aside, it is common to use bits-per-byte as a loss metric instead of the per token calculation, precisely because of the effect of different tokenizers.
Re:Kimi K3 is now live by smallerize on Thursday July 16, 2026 @02:05PM (#48938014)
It's going to vary dramatically based on which text you put in. Really it's hard to make one benchmark number that's relevant to all cases. But maybe we can make something a little more specific, like regular English text, code, the model's own thinking tokens, image inputs etc.
Re:Kimi K3 is now live by kevincox on Thursday July 16, 2026 @03:06PM (#48938839)
But even that isn't the whole story because the models can produce wildly amount of thinking output as well as regular output for a similar query. Sometimes you can take a cheap model and have it think a ton or an expensive model that thinks little and get similar results. But the number of tokens generated will be wildly different.
Re:Kimi K3 is now live by semiquaver on Thursday July 16, 2026 @02:27PM (#48938286)
I’ve been struggling to understand the reason for the newer apparently less efficient Anthropic token encoding. If all inputs are less efficient in this encoding, why does it exist? Has Anthropic released any information that would convincingly show it was anything other than a stealth price hike? Please don’t respond if you are speculating.
Re:Kimi K3 is now live by remus on Thursday July 16, 2026 @02:36PM (#48938407)
> Please don’t respond if you are speculating.

I doubt you are going to get a response from an anthropic employee, but I think it is safe to assume they have swapped to a new tokenizer because it improves the performance of their models.

Re:Kimi K3 is now live by re-thc on Thursday July 16, 2026 @03:09PM (#48938879)
> the reason for the newer apparently less efficient Anthropic token encoding

Less efficient in token usage but per the blogs; it enables the model to perform better.

Re:Kimi K3 is now live by mnicky on Friday July 17, 2026 @03:35AM (#48944366)
It's really simple I think.

More tokens per same text length means more capacity to encode information. More information means model can potentially perform better.

They introduced it around the time the Mythos came so my speculation is that if you have more capable model at some level you may find the current information encoding not using its full potential.

We will see whether OpenAI also introduces new tokenizer when they come to Mythos-size models.