Can we stop this kind of misinformation? Training a model to map a token to individual letters are no harder than training a model to be fluent at English. Arithmetics with small number of digits are achievable as well. You can just try a small 7B model yourself. If you don't know where to start, try the mistral instruct v0.2, and this is how it goes,

> [INST] Spell out the following word letter by letter: margorczynski [/INST] m - a - r - g - o - r - c - z - y - n - s - k - i

> So, the word "margorczynski" spelled out letter by letter is: m-a-r-g-o-r-c-z-y-n-s-k-i.

The text between `[INST]` and `[/INST]` is the input. The text after `[/INST]` is the output.

Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs?

https://twitter.com/karpathy/status/1657949234535211009

I'm not arguing that you can't use single chars just that many of the issues parent discussed are caused by this.

The easy solution is to create an additional dataset that is token aware. I.e. you take 1% of the dataset and take random tokens and split them into smaller tokens while expecting the same answer on the character level. This should force the model to learn multiple token representations of the same character strings.

Letter-by-letter tokenization increases inference and training costs and latency (as you need more tokens)

But they should be able to select the correct token for homophones, which amounts to the same thing.