OpenArch – PyTorch implementations of modern LLM architectures

Re:OpenArch – PyTorch implementations of modern LLM architec... by tweedler290 on Monday September 14, 2026 @09:38AM (#49696683)

This is really cool!

Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?

[ Parent | on HN ]

Re:OpenArch – PyTorch implementations of modern LLM architec... by anuj0456 on Monday September 14, 2026 @09:44AM (#49696778)

thanks for pointing out. its a bug

[ Parent | on HN ]

Re:OpenArch – PyTorch implementations of modern LLM architec... by tweedler290 on Tuesday September 15, 2026 @07:47PM (#49720399)

ur most welcome! if you're curious how I found it, I have made a code intelligence system. I literally fed it your repo and asked for bugs. You can try it yourself here: https://benzi.fly.dev/

[ Parent | on HN ]

Re:OpenArch – PyTorch implementations of modern LLM architec... by anuj0456 on Monday September 14, 2026 @10:19AM (#49697320)

fixed

[ Parent | on HN ]