This paper and other similar works changed my opinion on that quite a bit. It shows that to perform text prediction, LLMs build complex internal models.

https://news.ycombinator.com/item?id=38893456