Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)

The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!

the most obvious example was counting the Rs in strawberry.

figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together

theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely

you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it