Agreeed. As a non-ML developer, I think this is the paragraph that sells the work, even if it may be an oversimplification:

We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs

I think it says something very interesting about human language and information processing in general.