> I would think you could create an incredibly lean and smart "just React and React Native" model.

You can, but it's not as useful as you might think.

It needs to at least understand 1 human language to understand your intent to implement features.

If GRAM turns out to be a 5000x multiplier for local reasoning, you could theoretically train a 500M parameter model on just a programming language to understand stack traces to fix bugs and be incredibly powerful.

But most people also want it to understand human language to implement features as well.

Because then it can't just understand React and JavaScript - it needs to understand thousands of commonly used dependencies, the DOM, CSS, HTML, etc...

And for that you need A LOT more parameters than you might expect.

You can definitely get a ~3B active parameter model that can run comfortably on today's hardware to be VERY good at coding once all of the SOTA architectures are added to a single model - especially if we get better tool calling to give models better context per language.

You could get 100x performance if you feed the models ideal context... So a 3B model today can perform almost as good as ~300B model if you give it really good context vs flood it with mostly garbage it doesn't need across your repository.

We just want it to understand how to write code. We don’t also need it to know how to grow a potato.

The disjoint set of English related to strictly growing potatoes and adding features to code is a lot smaller than you probably think...

It is hard to cut out a huge portion of English and truly understand English and human language.

You're just not saving as much as you might assume you could.

I think perhaps you misunderstand how much of being an effective coder is understanding business domain enough to not be constantly asking for clarification (or if one is a fool or an ai, assuming wrong answers). I reckon a vast collection of trivia on the level of knowing how to grow a potato is important for a programmer

And you can't know ahead of time, when you're training the model, what business domains it will be used for. Someone may decide to use it to optimize the watering and fertilizer cycles of their automated potato-growing setup, and suddenly the "how to grow a potato" texts that went into training the model are actually the very things that make the difference between success and failure for the code the model spits out.

To me, the magic with LLMs has always been on the input side. It needs to understand what you mean in order to do what you ask. Most people are pretty terrible at communication, and general world knowledge seems to help with that.

... unless the software is potato farm software.

Programming is not a rare skill, the interaction with domain knowledge is.

"Make a React app to run my coffee shop" requires knowing React but also knowing what a coffee shop is.

Only if you're going after the "vibe coders" audience. Regular developers would be fine with a lightweight local llm capable of scaffolding and wiring a dozen of bog-standard components in a few lines of natural language.

Sure but it doesn't need to know everything.

It doesn't need to know different languages, every programming lanuage and co.

We will for sure get to this in the comming years. After all they will have to start finetuning their traning data anyway

The syntax is the easier part - most programming tasks require the reasoning and understanding of a large world model to solve problems.

Fine tuning a 'lean and smart' model works really well for discrete, repeatable high volume tasks like support ticket triage, lead classification, content filtering, labelling, generating content with a voice, etc.

Inefficient token burn by throwing large models at everything is definitely a problem - it's like hiring Phd's to answer the phone or to wash dishes.