But the truth is that very few people are actually trying to understand LLMs. Rather, what the vast majority do, is that they poke them unsystematically and then ooh and aah about all the things they can do that earlier language models couldn't do.
Careful analysis? Thorough, systematic experimentation? Theoretical results???? I have yet to see any of that and of course it's a joke to talk about theoretical results when it comes to machine learning research, these days.
It's as if we don't have any computer science, any theories of information, complexity, computation, learnability, etc, that we could muster to try and explain how a complex computer program works. So are people really trying to understand or are they just happy to ride the hype train? Because I obviously think it's the latter.
Maybe you are not impressed by the answer to "how does it work?", but we need to think about "what does it learn?". What data, what skills, what limitations. It's not just any data, it is our whole culture as a dataset, does it have some kind of special properties?