There are structural limitations (i.e. this implementation never looks at preceding context and sometimes that matters, nor does it understand other clues like punctuation or multi-word suffixes). Nevertheless, the accuracy is high enough that I'm not sure it'd be easy for a small model to beat it, especially not without considerably more work to make sure your inputs cover more context (which in principle the plain statistics approach could likely deal with too).
If the whole point of multi-layer networks is to deal with weirdly shaped, non-obvious manifolds in a very high-dimensional space, then this problem just doesn't look that difficult and perhaps does not need that mathematical finesse: just store the prototypical examples and you're pretty much there without anything fancier.
It doesn't need to be a full-fledged LLM either. What I mean is that a probabilistic machine learning model trained on a sufficiently large dataset is probably a better choice here. Small language models are pretty efficient nowadays, even for on-device use cases with constrained resources.