Not sure the impact on kvcache though.
then there was a fully looped model (forget the name) where all the layers were looped (which is an obvious mistake as it's known LLM's have token processing and reasoning stages) and showed some decent results.
and now the concept worked itself into the mainstream.
If you're curious, it's a very cool paper! https://arxiv.org/html/1807.03819v1
The Show HN of someone applying this recently, through layer duplication: https://news.ycombinator.com/item?id=47431671
That's just a general simplification though. One of the key motivators in this general area of research is adaptive latent reasoning, meaning in theory, some tasks only require 2 loops and some require 8.
There's a lot of different approaches here and many of them approach the problem from different traditions. Some looped transformers use a fixed loop count, others follow after deep equilibrium models and iterate towards a fixed point or some other stopping mechanism.
Some fun reading if you're interested in this field:
- TinyRecursiveModels - Blew up a couple of years ago when it performed really well on ARC AGI 1 using a comparatively small model with a large task-level table of embedding prefixes. https://github.com/samsungsailmontreal/tinyrecursivemodels
- Sebastian Raschka's piece on the general architecture patterns we see today: https://sebastianraschka.com/llm-architecture-gallery/looped...
- It lets the model do more of its reasoning without emitting tokens. This is helpful both because a) insanely long token streams are annoying, but more importantly b) long reasoning streams eat context. It's not a direct performance enhancement, since each token is more expensive to generate (you need to loop!), but it reduces spam in the context window which likely translates into some degree of intelligence win for multi-turn since the model won't have to sift through as much noise, and it'll need to compact less frequently per task which is also likely an indirect intelligence win and an indirect cost/perf win per task: post-compaction re-reads of files are basically useless spam to an inference engine, so if you could do the entire task within one context window, you'd hit the engine less per task.
- It lets potentially every token use more computation, instead of keeping the test-time-compute scaling purely in the upfront CoT domain. While it's helpful to think upfront, it can be useful to think mid-stream too.
Overall it's potentially much better.
Looping is controversial among the alignment/safey crowd, because the more models can think without printing a trace, the less understandable they might be to us.
But instead of duplicating the layer's weight, you run through them multiple times at runtime.
To me, this means they probably won't.
And then I test it in comparison and I remember Goodhart's Law.
This model, for sure, is interesting. But there's more than one way to peel a potato, and again and again I find that in practice these benchmarks don't really seem to mirror normal "coding assistant" uses.
I think we’re going to start seeing a lot more of these types of advances, especially on smaller models
I’ve been able to graft reasoning and math problem solving into a small, locally trained model with 6.5M params. And I’ve also been able to add abilities or improve them by “growing” the mode to 8.5M params (this usually means redistributing the weights in a slightly different network size, then finetune the new size)
The model is so small that I can sometimes run dozens of experiments per day. There are a lot of papers and new models coming out. A lot of the stuff they are doing at 100B-1T still works at smaller scale (usually not the scaling inference-related optimizations, but rather things like weight programming, or a “looped” architecture). It’s a lot of fun to run at home on my laptop
I imagine bigger labs with bigger budgets (more GPUs) are mostly focused on 20B-1T+ models. But there’s a lot to explore in local “home models”
It would be great if it was true.
Running a local frontier model is the dream.
…but, its been obvious since the beginning that this is a fantasy without a paradigm shift.
People want general purpose models. People want parallel models. People want fast models.
This is interesting, because it does represent a paradigm shift, perhaps… but most people saying “Ive been expecting this…” have simply not had a realistic understanding of complexities involved.
Personally; Im skeptical. This is what people were saying about 1 bit models. That turned out to make very little meaningful diff and did not result in a local hosting utopia.
I guess we will see… I feel like we will see slow incremental improvements in this space.
Sub 1B coding models remains a fantasy for now.
For sure, and it will continue to be so, until it isn’t
In the meantime, I think there’s a lot of encouraging developments
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkin...