blackcat201

I think one crucial part that everyone seems to have missed is that they replace the previous Gated MLA layers (Kimi K3) with DSA (K3.1) via post-training to reduce the inference cost. In theory it should lower the kv-cache pressure by a huge magnitude scale.

who says they aren't benchmaxxing?

I think the author need to rethink out of the box what's LLM routers in a traditional sense ( input in, route, output ) and move to think how a router would work in agentic workflow. See cognition Devin Fusion design.

Shameless plug, for anyone who's interested in "self-improvement" agent check out StreamBench[1] where we benchmark and try out what's essential for improvements in online settings. Basically we find feedback signal is vital and the stronger the signal the more improvement you can get if you were able to feed it back to the agent in terms of weights (LoRA) or in-context examples.

[1] https://arxiv.org/abs/2406.08747

Do beware on some reasoning task, our recent work[0] actually found it may cause some performance degradation as well as possible reasoning weakening in JSON. I really hope they fix this in the latest GPT-4o version.

[0] https://arxiv.org/abs/2408.02442

The standard operation is to stop and check if any machine was out of calibration. So yes

I own my LLM not because I need it now but having the luxury to fall back if openai ran out of money

I have been following the vector database trend back in 2020 and I ended up with the conclusion: vector search features are a nice to have features which adds more value on existing database (postgres) or text search services (elasticsearch) than using an entirely new framework full of hidden bugs. You could get way higher speedup when you are using the right embedding models and encoding way than just using the vector database with the best underlying optimization. And the bonus side is that you are using a stack which was battle tested (postgres, elasticsearch) vs new kids (pinecone, milvus ... )

This looks pretty interesting! But the landing page has only one sentence : Understand and implement research papers faster, followed by a get in touch button. Care to extrapolate more? The blog button doesn't work as well?

Note that stability have been funding freelance researcher by providing compute resource such as RWKV[1], Open Assistant, some works by LAION[2] and lucidrains[3]

[1] https://github.com/BlinkDL/RWKV-LM

[2] https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K

[3] https://github.com/lucidrains/gigagan-pytorch#appreciation

On the other hand, meta AI research should be renamed as OpenAI. They are the only few big institute who open almost every models they train (galactica, OPT, M2M, wav2vec ...)