In that Tencent study less common languages got easier problems. Which explains why racket also performed suspiciously well in it. The authors recognize the problem themselves. I like elixir, but it's imo embarrassing how often elixir people cite that study despite it not actually supporting what they claim.

A much more recent and imo more realistic comparison of different languages with ai, is openai's study reimplementing sqlite in different languages: https://openai.com/index/low-resource-programming-languages/

That showed about what we expect, that popular languages like python, javascript, java, C#, etc do best, while less popular languages like Common Lisp, Ocaml, Zig, etc do slightly worse. But the difference is small enough I'd still just choose whatever language and ecosystem I know best that is appropriate for a project.

I would definitely be skeptical of a study that finds using an LLM/agent to write sqlite in JS being better than writing in Zig, but it strongly depends on what “better” or “worse” mean.

It means how much of the sqlite test suite it passed. They were just measuring correctness, not performance or memory consumption, where of course Zig should win.

But the aim of that experiment was measuring correctness of llm written code in different languages, so used sqlite as a benchmark considering it's fairly complex software with a comprehensive test suite. It is not suggesting that because a language scores better on that benchmark it'd be better suited to writing the next sqlite, due to obvious other factors like performance...

There’s just not enough information in the “study” to be helpful or informative though. It’s just a list of numbers with not enough methodology to reproduce them, so to me they’re no more meaningful than a press release.