News for hackers. Stuff that matters.

A comment on The FFT Strikes Back: An Efficient Alternative to Self-Attention. Parent comment

Re:The FFT Strikes Back: An Efficient Alternative to Self-At... by topwalktown on Friday February 28, 2025 @12:23AM (#43202001)
Transformers like Llama use rotary embeddings which are applied in every single attention layer

https://github.com/huggingface/transformers/blob/222505c7e4d...

[ Parent | on HN ]
Re:The FFT Strikes Back: An Efficient Alternative to Self-At... by markisus on Saturday March 01, 2025 @08:04AM (#43218856)
Very interesting! Do you know if there were any studies about whether this improves performance?
[ Parent | on HN ]