they mention similar performance to vanilla transformer with significantly reduced param count though
Re:Differential Transformer
by karmasimida on Tuesday October 08, 2024 @12:08PM
(#41778916)
I mean it doesn’t necessarily needs 2x QK to match that performance, in terms of accuracy, of a regular transformer right?
For one, speed and memory. They have twice as many Q and K weights in the attention blocks, leading to a ~10% reduction in throughput on their H100 (table 7 in appendix A).