Re:Understanding Positional Embeddings in Modern LLMs
by mathisfun123 on Sunday November 15, 2026 @02:46PM
(#50510012)
> As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks
It's always like this because people can't admit ML is just spray and pray. So they superimpose theory.
Re:Understanding Positional Embeddings in Modern LLMs
by programjames on Sunday November 15, 2026 @03:29PM
(#50510379)
Most people who publish papers in ML are alchemists, but please do not assume all of us are. The original authors of the RoPE paper knowingly designed their embedding to be translation-invariant. I do not know if they were aware it was a group representation, but what they were trying to design was in essence a group representation.
Re:Understanding Positional Embeddings in Modern LLMs
by fc417fc802 on Sunday November 15, 2026 @04:09PM
(#50510786)
Both things can be true and useful. You can search for solutions in a fairly arbitrary manner, and when you find something that works you can attempt to develop a theory to explain it in the hope that said theory will make future search efforts more efficient.
Re:Understanding Positional Embeddings in Modern LLMs
by Mabusto on Sunday November 15, 2026 @04:29PM
(#50510992)
This is the most frustrating part of trying to understand machine learning. You can read a bunch of really great math books that rigorously build concepts from the ground up (calculus, linear algebra), then you start to get into the more hand wavey math (_some_ statistical concepts feel like this) and then you start getting into some of the theory behind machine learning systems and expecting the same amount of rigor and ability to understand _why_ something works and it’s always an unsatisfying explanation. I really wish there would be more disclosure like “this is what we tried and this seems to work, we think this is why” rather than present the method, give some mathematical formalization of the method, then go “it works great” and then move on. It leaves people like me wondering if there is something that I’m missing or some hole in my knowledge that is obvious to everyone else, OR does no one actually know why this works and just don’t want to say. Just give us a hint, make the shrug emoji or something, the more I learn about machine learning, the more I get that it really is more art than science.
Re:Understanding Positional Embeddings in Modern LLMs
by programjames on Sunday November 15, 2026 @05:12PM
(#50511400)
Most people who write ML papers are at the stage of alchemy—try a bunch of things and see if you can create gold out of lead. Usually their papers are not worth reading, because even if they do something right, they do not know what made it right, and it cannot lead to further or generalizable insights.
A few ML researchers do actually have a lot of this theoretical knowledge. You can see this in papers like Variational Diffusion Models, Continuous Normalizing Flows, <?> Delta Attention. These papers are usually also better written, because the authors understand why things work so they can deprioritize irrelevant details.
Re:Understanding Positional Embeddings in Modern LLMs
by nsagent on Sunday November 15, 2026 @06:32PM
(#50512032)
Ali Rahimi's NIPS 2017 Test of Time Award speech lays out that case especially well:
https://youtube.com/watch?v=x7psGHgatGM
I presume your comment is in reference to this speech, as he famously states: "Machine Learning has become alchemy" in it. For anyone who has some time and hasn't watched it, I'd highly recommend it.
Re:Understanding Positional Embeddings in Modern LLMs
by D-Machine on Sunday November 15, 2026 @04:39PM
(#50511083)
This is exactly correct.
Re:Understanding Positional Embeddings in Modern LLMs
by esafak on Sunday November 15, 2026 @06:30PM
(#50512016)
There is no enlightenment in this. Are you going to make the next breakthrough by throwing yet more mud at the wall, or are you going to take educated steps based on a "post-factum" analysis of what worked well so far?
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.