But by the standard machine learning definition (performance on train/test split), LLMs generalize extremely well. This isn't even in doubt.

They do generalize. The claim is that the fine details are not the result from generalization, but from repeating test data verbatim. That seems consistent both with my intuitive understanding of neural networks and with the behavior I've observed, so I'm inclined to agree. So what does that mean? It means that while LLM:s can produce impressive output, the most impressive results that people are touting probably have a significant amount of verbatim training data in them. In other words, they're good but not as good as they seem to be.

Mmm the most impressive thing I see LLMs do is take a piece of unstructured input and transform it in some way - summarize, extract information as JSON, etc. This wouldn't be possible if it were repeating training data verbatim, since it works on new novel inputs.

Some of the most impressive things I've seen LLMs do is take a grammar in, in a format that does not strictly conform to any specific variant of a formal grammar notation, then generate output that conforms to that grammar, and reason about why it conforms, and how.

Most people would struggle immensely with a task like that even if handed a textbook on the subject, and no amount of training data happens to contain text in or about a language governed by a grammar of random nonsense I fed in.

There are areas where their reasoning is really awful. Ironically that's often when they seem most human-like. E.g. I just had a lengthy "argument" with ChatGPT over comparing the theoretical computational power of Markov Decision Processes vs. Turing Machines with various assumptions about the decision maker in the MDP, and it's reasoning was riddled with logical fallacies that I could very well see a high school students confronted with trying to compare the two based on a Wikipedia level of understanding of either without sufficient understanding to reason about how different aspects can be made to model the other.

But there are plenty of areas where you can get them to produce good results where the "fine details" could not possibly be repeated verbatim from the test data because they didn't exist prior to the conversation.