The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated.
Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.
Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
Another slept-on feature is the ability to quickly write your own static analyzers, which along with the myriad of existing analyzers out there, provide excellent guard-rails to prevent slop.
And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.
Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
Think of it as being like a road trip. Arrival is binary. You have either arrived or have not arrived, but distance can tell you how close you are to arrival. Being almost there does not imply that you have arrived, however. Same applies here. Some samples were closer to being correct than others, but none were correct. They were all incorrect.
But as you alluded to earlier, I don't suppose anyone wants code that fails to solve the stated problem. Given that we have empirical evidence that LLMs cannot produce correct code within a given set of problems (and I suspect that extends to most problems), correctness is a weak signal.
What is a useful is to know is how much additional effort is required to make the program correct. That is what quality has traditionally meant as it pertains to code. High quality codebases are considered high quality because the effort to reach and maintain correctness is considered to be low. We do not have enough context to know for certain if that is what was meant in the earlier comment, but it seems likely. What we do know is that the comment is about quality while the article is about correctness.
Yes, quality is what the earlier commenter is talking about. Yes, idiomatic code is often considered to be what makes code high quality because idiomatic code is believed to make it easier to reach correctness (easier to read, easier to reason about, easier to test, etc.) than if you have to work with "spaghetti", which is considered low quality because it makes it much harder to ensure the code is correct. Quality exists on a gradient. There exists a full spectrum between a complete spaghetti monster mess and perfect architected idiomatic code. This we both read the same way it seems.
Whereas the article is about correctness. The program is either correct or not. Distance was used in the article to share how far away a given program was from being correct. But distance and quality not the same. This is a completely different way to look at software as compared to the topic of quality, as we already established at the beginning of our discussion.
We mostly avoid writing assembly because it isn't portable. So-called "portable assembler" is still very popular. One of the most used languages out there. That has little to do with the quality of codebase, though. Focus, my man.
My experiences are of course anecdotal, but if you have some other strong empirical data to show, I'd love to see it.
For existing codebases I made the experience, that LLMs are very good at replicating their style.
At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous.
Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.
However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.
Add in golang doesn’t have proper types for Option<T> to have a nilable type. Its pointers and null pointer bugs are plenty, no proper stack traces.
I find myself to be less productive.
For actual output quality, the main deciding factor is simply how much tooling it is there for the LLMs to check their own work, as LLMs seemed to avoid using a lot of libraries in general. That's why C# is underrated due to the tooling strength of the .NET ecosystem, as long as you tell LLMs to avoid using reflections unless absolutely necessary.
C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.