News for hackers. Stuff that matters.

A comment on The lifecycle of a code AI completion. Parent comment

Re:The lifecycle of a code AI completion by sqs on Sunday April 07, 2024 @10:55PM (#39965825)
The list of supported languages is at https://sourcegraph.com/docs/cody/faq#what-programming-langu.... It works for all programming languages, but it works better on some than others, depending on the LLM and the language-specific work we've done on context-fetching, syntax-related heuristics, etc. On the LLM point, Cody supports multiple LLMs for both chat and autocomplete (Claude 3, GPT-4 Turbo, Mixtral, StarCoder, etc.), which is great for users but makes it tough to give any formal definition of "supported language".
[ Parent | on HN ]
Re:The lifecycle of a code AI completion by nicklecompte on Monday April 08, 2024 @08:49AM (#39969170)
Even that link doesn't claim to be comprehensive, including an extremely vague "shell scripting languages (like Bash, PowerShell)," as if I am supposed to cross my fingers and hope this means a nonstandard shell like fish works, when it seems like that could introduce weird or inscrutable bugs coming from incompatible bash syntax. (One under-discussed downside of LLM code generation is that it further penalizes innovation in programming languages.)

What I find frustrating is that Cody is doing deterministic language-specific tricks that don't depend on the underlying LLM, so why not just give a list of all the languages you did deterministic tricks for and call that the supported languages? Why be vague? Why deflect to the underlying LLM to encourage people to try your product with languages you won't do any work to support?

Claiming it works on "all programming languages" is lazy and dishonest, and clearly false. There's nothing magical about LLMs that relieves software developers of the duty to specify the limitations of their software.

[ Parent | on HN ]
Re:The lifecycle of a code AI completion by sqs on Wednesday April 10, 2024 @04:24PM (#39995338)
The code is Apache 2 and we linked it from the blog post. You can see all of the tricks!

Nothing here is dishonest. We’re open about the limitations of the LLM being the main determining factor, and the numerous languages we’ve prioritized making it work well on. I see your point about not mentioning “etc.” in the shell scripting parenthetical. Will update.

[ Parent | on HN ]