News for hackers. Stuff that matters.

A comment on Kimi K3 Architecture Overview and Notes. Parent comment

Re:Kimi K3 Architecture Overview and Notes by itkovian_ on Tuesday July 28, 2026 @04:51PM (#49089738)
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
[ Parent | on HN ]
Re:Kimi K3 Architecture Overview and Notes by ModelForge on Tuesday July 28, 2026 @06:56PM (#49091116)
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
[ Parent | on HN ]
Re:Kimi K3 Architecture Overview and Notes by cma on Wednesday July 29, 2026 @06:27AM (#49095577)
The paper says:

"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."

[ Parent | on HN ]