From your link:
> for every doubling of model size the number of training tokens should also be doubled
Well, I don't think many claim that increasing the model size while also increasing the training tokens doesn't make sense.
The problem is how to get the training tokens...
Interestingly enough labs do train with 100s or even 1000s times more tokens than what's proposed by the paper as an optimal value.
And the answer to your question is basically just “synthetic data”.
The Chinchilla paper is probably the poster child of “paper that aged like milk” in LLM field though. Everyone over the past 4 years have trained with several orders of magnitude more data than the Chinchilla optimal number.
If you don’t know which data has the highest value, though more at it. Chinchilla is probably still true, but with that caveat. And it could be like advertising… you know a lot of it is worthless, but you don’t know which part.
(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)