Re:From Multi-Head to Latent Attention: The Evolution of Att...
by mrtesthah on Saturday August 30, 2025 @03:05AM
(#45072533)
Do we know if any of these techniques are actually used in the so-called "frontier" models?
[ Parent
| on HN ]
Re:From Multi-Head to Latent Attention: The Evolution of Att...
by vinithavn01 on Saturday August 30, 2025 @03:17AM
(#45072588)
The model names are mentioned under each type of attention mechanism
Re:From Multi-Head to Latent Attention: The Evolution of Att...
by gchadwick on Saturday August 30, 2025 @06:10AM
(#45073417)
Who knows what the closed source models use but certainly going by what's happening in open models all the big changes and corresponding gains in capability are in training techniques not model architecture. Things like GQA and MLA as discussed in this article are important techniques for getting better scaling but are relatively minor tweak vs the evolution in training techniques.
I suspect closed models aren't doing anything too radically different from what's presented here.
Re:From Multi-Head to Latent Attention: The Evolution of Att...
by zackangelo on Saturday August 30, 2025 @01:23PM
(#45076391)
Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.