News for hackers. Stuff that matters.
Re:Understanding Positional Embeddings in Modern LLMs by mathisfun123 on Sunday November 15, 2026 @02:52PM (#50510070)
This is just basic lie theory shoe-horned where it doesn't belong lol.
Re:Understanding Positional Embeddings in Modern LLMs by programjames on Sunday November 15, 2026 @03:38PM (#50510455)
It is called representation theory, and once you have seen even the basics of representation theory it is obvious the authors of the original RoPE paper were trying to rederive a representation for the translation group.

The reason you want to know the general theory behind positional embeddings is it lets you work with other kinds of relative positions. For example, quad/octtrees are useful for faster attention on large images/videos, or a Gaussian splat might want 3D translation + rotation invariance.

I believe you can also go further and start adding in gauge groups and whatnot, but I just thought about that off the top of my head.