News for hackers. Stuff that matters.
Re:Understanding Positional Embeddings in Modern LLMs by Mabusto on Sunday November 15, 2026 @04:29PM (#50510992)
This is the most frustrating part of trying to understand machine learning. You can read a bunch of really great math books that rigorously build concepts from the ground up (calculus, linear algebra), then you start to get into the more hand wavey math (_some_ statistical concepts feel like this) and then you start getting into some of the theory behind machine learning systems and expecting the same amount of rigor and ability to understand _why_ something works and it’s always an unsatisfying explanation. I really wish there would be more disclosure like “this is what we tried and this seems to work, we think this is why” rather than present the method, give some mathematical formalization of the method, then go “it works great” and then move on. It leaves people like me wondering if there is something that I’m missing or some hole in my knowledge that is obvious to everyone else, OR does no one actually know why this works and just don’t want to say. Just give us a hint, make the shrug emoji or something, the more I learn about machine learning, the more I get that it really is more art than science.
Re:Understanding Positional Embeddings in Modern LLMs by programjames on Sunday November 15, 2026 @05:12PM (#50511400)
Most people who write ML papers are at the stage of alchemy—try a bunch of things and see if you can create gold out of lead. Usually their papers are not worth reading, because even if they do something right, they do not know what made it right, and it cannot lead to further or generalizable insights.

A few ML researchers do actually have a lot of this theoretical knowledge. You can see this in papers like Variational Diffusion Models, Continuous Normalizing Flows, <?> Delta Attention. These papers are usually also better written, because the authors understand why things work so they can deprioritize irrelevant details.

Re:Understanding Positional Embeddings in Modern LLMs by nsagent on Sunday November 15, 2026 @06:32PM (#50512032)
Ali Rahimi's NIPS 2017 Test of Time Award speech lays out that case especially well:

https://youtube.com/watch?v=x7psGHgatGM

I presume your comment is in reference to this speech, as he famously states: "Machine Learning has become alchemy" in it. For anyone who has some time and hasn't watched it, I'd highly recommend it.