> Today LLMs have not been demonstrated to be capable of synthesizing novel code.

They are capable of doing that (to some extend). Personally, I've generated plenty of (working) code to solve novel problems and I'm 100% sure that code wasn't part of the training set.

That's interesting. What kind of problems?

I’ll second that. A simple example is asking it to write pyplot or tikz code to draw maps and pictures. I got it to draw a correct floor plan for the White House entirely with python code. It amazes me that it understands spatial layouts from training only on text such that it can draw physically accurate diagrams, and it understands graphics libraries well enough to draw with them. Apparently predicting text about spatial locations requires an internal spatial map. Thinking about the chain of understanding of different concepts that have to be integrated together to accomplish this shows it’s not a simple task.

> It amazes me that it understands spatial layouts from training only on text such that it can draw physically accurate diagrams, and it understands graphics libraries well enough to draw with them.

Is there evidence of this? The Whitehouse floor plan is very well known, and available online in many different formats and representations. Transforming one of those into a sequence of calls would be easier.

Have you tried this with a textual description of a building that does not have any floorplans available, i.e. something unique?

To be fair, most coding problems are not novel problems.

I’ll be more impressed if chatGPT can take a large, poorly maintained, poorly documented codebase and make meaningful changes to it.

Far easier to write something from scratch than to modify something that was first written in Fortran in 1991.