Not OP, but I have been poking around with LLMs for a year and I'd like to add to the conversation.

In my experience, LLMs are word predictors, and the impacts of that fact are not immediately obvious.

LLMs are capable of "explaining" what code does. What it is doing under the hood is pattern matching: I've seen code that looks like X, with an explanation that looks like Y

LLMs are capable of formatting text. It has seen English written like X, that is reformatted to look like Y

One resounding fact my team has found over and over is that "the things we think are hard for LLMs aren't necessarily hard; the things we think are easy aren't necessarily easy"

Worth noting that pattern matching and term rewriting are a sufficiently general combination that it forms the basis for Mathematica.

Trouble sneaks in when the pattern matching is only correct most of the time. Eg, if some code for regex-based search missed anywhere from 0.1% to 10% of matches, with the miss rate depending on the regex and no obvious way to know which regexes have worse miss rates, the utility of your regex-based search would be limited. LLMs are like this, but their generality makes them useful in spite of this limitation.

I would just add that Mathematica-style term rewriting (e.g. analytical integration, or equation solving) is done with _semantics-preserving_ symbolic solvers, which are hand-made and human-reviewed to guarantee correctness.

LLM style pattern matching and rewriting does not preserve semantics, except accidentally due to an overwhelming amount of examples.