And the fact that they trained it as a looped model in 4-bit native is also kind of impressive to me as well, because it combines recurrence, routing, and narrow-format - all of which adversely impact numeric stability during training.
Apparently trained on a ~couple dozen trillion tokens; which is far smaller than most pretraining and posttraining runs.
On the other benchmarks it trades blows or beats Qwen; as Thinky points out, looping helps reasoning but doesn't scale memorization, so tasks requiring broad world knowledge without tools/search won't be helped by looping.
Still super impressive, and probably replaces Qwen 27B in pretty much any workload!
Which is counter to the way most open weight models have been leaning. Many current models are MoE (Mixture of Experts) which by their very nature are parameter heavy. They are trading parameter efficiency for compute efficiency which is kind of the wrong tradeoff for current consumer hardware like gaming GPUs.
It depends, people with Macs or AMD with unified memory are more constrained by memory bandwidth than with resident memory.
Unfortunately for us, looping doesn't help on that from.
We already know from official model cards that the GPT-6 models are substantially less monitorable than other (non-looped) models. Which is not surprising, since they are doing some amount of Neuralese thinking between each token.
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.