It would be great if it was true.
Running a local frontier model is the dream.
…but, its been obvious since the beginning that this is a fantasy without a paradigm shift.
People want general purpose models. People want parallel models. People want fast models.
This is interesting, because it does represent a paradigm shift, perhaps… but most people saying “Ive been expecting this…” have simply not had a realistic understanding of complexities involved.
Personally; Im skeptical. This is what people were saying about 1 bit models. That turned out to make very little meaningful diff and did not result in a local hosting utopia.
I guess we will see… I feel like we will see slow incremental improvements in this space.
Sub 1B coding models remains a fantasy for now.
I think we’re going to start seeing a lot more of these types of advances, especially on smaller models
I’ve been able to graft reasoning and math problem solving into a small, locally trained model with 6.5M params. And I’ve also been able to add abilities or improve them by “growing” the mode to 8.5M params (this usually means redistributing the weights in a slightly different network size, then finetune the new size)
The model is so small that I can sometimes run dozens of experiments per day. There are a lot of papers and new models coming out. A lot of the stuff they are doing at 100B-1T still works at smaller scale (usually not the scaling inference-related optimizations, but rather things like weight programming, or a “looped” architecture). It’s a lot of fun to run at home on my laptop
I imagine bigger labs with bigger budgets (more GPUs) are mostly focused on 20B-1T+ models. But there’s a lot to explore in local “home models”