Re:Understanding Positional Embeddings in Modern LLMs
by nsagent on Sunday November 15, 2026 @06:32PM
(#50512032)
Ali Rahimi's NIPS 2017 Test of Time Award speech lays out that case especially well:
https://youtube.com/watch?v=x7psGHgatGM
I presume your comment is in reference to this speech, as he famously states: "Machine Learning has become alchemy" in it. For anyone who has some time and hasn't watched it, I'd highly recommend it.
A few ML researchers do actually have a lot of this theoretical knowledge. You can see this in papers like Variational Diffusion Models, Continuous Normalizing Flows, <?> Delta Attention. These papers are usually also better written, because the authors understand why things work so they can deprioritize irrelevant details.