The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining.
The reason byte level models aren’t as popular is because inference speed is significantly slower for a given completion length, and training for the same context window in characters is significantly more expensive
I say this as someone who’s deployed ByT5 vs normal T5 in prod, having found this to be true in real world use cases. No reasonable amount of fine tuning can save GPT from performing poorly on a space insertion task (“WHOLEFDSMKT” -> “Whole Foods Market”), while ByT5 just works.
I don’t know why people feel so strongly that the tokenization is a weakness, but ultimately there’s not much choice but to agree to disagree.
Agree that tokenization isn’t a weakness for most general applications, disagree that it isn’t a weakness for the specific string manipulation task that the blog post is referencing
How about the reply posted 5 minutes before yours. Appears like pretty good evidence.
https://news.ycombinator.com/item?id=33915690 even points out a counterexample demonstrating that it can do character level tasks just fine.
This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence.
It’s a bit like saying that you perform poorly on character manipulation tasks because you don’t read individual letters.
Biology analogies aside, I haven’t seen anything to suggest that utf8 level tokenization causes a significant decrease in perplexity across large datasets. (Note that the “large dataset” criteria is required. It’s certainly possible to demonstrate improvements in restricted cases, but no one is really interested in the restricted case unless you have a very specialized task. In which case, sure, specializations make sense — ChessGPT being an obvious example where tokenization just harms learning.)
So the tradeoff isn’t the large context window, but rather the desire to have a deep understanding of a massive amount of data. Specialized models will always have a place as a small component of the whole, but suggesting that this is a problem solved by superior architectures seems a little bit of a stretch.
I think what’s going on here is that OpenAI spent a lot of time giving feedback to their model about specific use cases, and ROT-13 was obscure enough (both in usage and in the data) that its performance is limited. I’d bet that if OpenAI did a few rounds of RL on this objective, the model would perform as well as its cousins.