That is 11 should be two tokens representing 1 and 10. That way the relationship can be more easily learned.
Otherwise it needs to memorize the relationship between every pair of numbers. Which is exponentially more difficult.
It's quite possible that this trade-off is worth it, that making math and poetry harder to learn isn't a big deal because of the savings elsewhere (models can still learn it eventually, as we have seen).
But BPE isn't exactly elegant. It's itself a learned encoding, trained on a different objective from the rest of the system. In the rest of the training, we let the GPT model decide how to assemble lower-level concepts into higher-level ones, and which information to ignore. On the lowest level we've hardcoded it for the model, and we know the model sometimes have to work hard to undo our encoding.
Having the token numbers sequential is meaningless and makes no difference to how a NN works. They are just pointers.
It needs to learn that the sequential value of the things pointed to by these tokens is important and it has only partially done that.
We see that in children learning their numbers too: As a small child "what comes after 39" (or some number) and you'll often get wrong answers.