Isn't it the case that for very short distances (several elements), power decay and exponential decay are (or can be made, with proper constants) more similar? Thus, if predictive models were originally studied only for very short sequences in the past (limited computational resources!), it seems to make sense that this is a mistake that anyone could have made more easily back then.
Would you consider LSTM an adequate model?
That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.
The relevant example from the paper:
I: Jane went to the hallway.
I: Mary walked to the bathroom.
I: Sandra went to the garden.
I: Daniel went back to the garden.
I: Sandra took the milk there.
Q: Where is the milk?
A: garden
What exactly those models are modelling?
From my non-professional perspective the above seems that it should have been very obvious - (and also that correlations between variables for natural languages would be better explained by multi-dimensional structure). That is if you told me that this were proved / formally supported as it is in this paper, my reaction would be a "that sounds like reasonable approximation" not - "that result sounds very surprising I must read the paper".
As someone outside of this field, it seems to me that this kind of result should have been very obviously foreseeable, hindsight bias and all that of course - but I would never have considered Markov processes to be an adequate predictability model for natural language. Though obviously the formalized results are important.
Could someone with more knowledge comment on what the current working assumptions were prior to this paper and what the consequences would be?