What's the finite group in this case?

I’m thinking the integers mod 2^n where n is something computers are good at (8, 32, 64). You have hardware support the group operation.

That is the traditional Fourier transform, except it can be a cyclic group of any size, doesn't need to be a power of 2. (Though FFTs with 2^n input size are particularly easy to implement.)

And it's not permutation invariant.

I was careless in my thinking, thanks for the correction. I was imagining since you sum the group elements in any order, there is a permutation invariance. But the group elements themselves play the role of the "token index" and group elements are not interchangeable. To actually make this idea interesting, one would have to use a group in which the choice of group element assigned to each input token would not affect the result.

You mean for the group operation to be standard modular addition? In that case (as a sibling comment says) you'll recover the classic discrete Fourier transform.

Not an expert in this space.

Aren't tokens transformed with position dependent information in most models?

I believe llama applies a rotation to the vector based on the position in the input.

That's true in the realm of LLMs. But even in this case, the position information is added only into the first layer. Tokens in later layers can choose to "forget" this information. In addition there are applications of transformers in other domains. See https://github.com/cvg/LightGlue or https://facebookresearch.github.io/3detr/

Transformers like Llama use rotary embeddings which are applied in every single attention layer

https://github.com/huggingface/transformers/blob/222505c7e4d...

Very interesting! Do you know if there were any studies about whether this improves performance?