News for hackers. Stuff that matters.
Re:You could have invented transformers from memory by melozo on Tuesday October 20, 2026 @01:48AM (#50161835)
Question to math people - are you actually able to intuitively understand and derive meaning from some of the formulae? How did you get to that point?

As an example - "The optimal linear memory store" talks about whitening the key by undoing key covariance through transposing and multiplying the "K" matrix, then inverting it, then multiplying it with K again. There must be some beautiful mathematical relationship there, but I'm looking at this completely confused as to how such a thing could effectively scale the key-space such that covariant keys are widened and avoid/reduce overlap.

I took linear algebra over a decade ago in college but obviously don't remember it. I'm just surprised that such semantically interesting outcomes can come of these things, but reading this for me as someone without that deep and second-nature understanding makes the math impenetrable for me. Even if I understand what gets multiplied by what and the theoretical basis for each operation, being able to read this and make sense of the math alone is impossible for me.

Re:You could have invented transformers from memory by applecoffeecake on Tuesday October 20, 2026 @05:47PM (#50173147)
The answer is that it just takes continuous and serious engagement with both general and specific mathematics typically through graduate-level courses and research degrees. The particular interpretation of least squares as noise whitening plus orthogonal projection is just a trick you would learn in an advanced statistics class. If you are not versed in the subject, you would not immediately understand the particular math here even as a mathematician! To demonstrate this point, at 56:00 in this video (https://youtu.be/VcUU42ZXnQs?si=kqkUWxnFv2lSa5FH), Fields Medalist Cedric Villani is confused by a similar remark about noise whitening and asks the speaker to explain what it means. Here, the speaker is applying the trick to the closely related concept of Linear Minimum Mean Squared Error (LMMSE) estimation which is the probability-theoretic counterpart to least squares. Moreover, Cauchy-Schwarz refers to a fundamental inequality that any mathematician would know, and would show the optimality of orthogonal projection. The general (Cauchy-Schwarz) gives you a foundation for understanding the specific (least squares theory).