kbwal7

Note that this sort of distillation is NOT for pre-training data (which is tens of trillions of tokens). I think the allegations against Chinese companies by Anthropic is more so that they distill SFT data (which is good for post-training, but you still need a strong base model)

I'm honestly not too aware of the true scene of which optimizers are used nowadays by which labs (though I think Moonshot has shown Muon's ability to scale to large models, and Muon is definitely more used now). That being said, obviously orthogonalization is pretty expensive! I looked it up and it's roughly 5-15% slower per step, but it converges in fewer steps fwiw. I'm guessing it's also less battle-tested than AdamW is, so AdamW still might be the safer option (and you're gonna have to use AdamW for your 1D params anyways).

I do think there are now open weight models that are on par with (or beating) Opus 4.5 by now (e.g. Kimi K3, GLM5.3). But yeah obviously the frontier closed source models seem to have pulled away once again, so open weight seems to be a few months behind right now (which might be too long to wait for a lot of people!).

Olmo is one truly open source model. https://allenai.org/blog/olmo3

I think it's just a harness that uses proprietary (or maybe IBM is hosting open weight?) models. They say on the site (in the FAQs) that you can't even choose which model!

So it seems to be a harness that auto-routes your queries with no way to change this.

It's pretty incredible how robust neural networks are to even things like architectural changes mid-run. The team at Meta when training their own copy of GPT-3 (called OPT-3) even changed optimizers mid-run (from AdamW -> SGD -> AdamW, https://arxiv.org/pdf/2205.01068 section 2.5)!