News for hackers. Stuff that matters.

A comment on Matrix Orthogonalization Improves Memory in Recurrent Models.

Re:Matrix Orthogonalization Improves Memory in Recurrent Models by hasley on Wednesday July 01, 2026 @11:23AM (#48748412)
I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix.

I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.

[ Parent | on HN ]
Re:Matrix Orthogonalization Improves Memory in Recurrent Models by aesthesia on Wednesday July 01, 2026 @11:53AM (#48748869)
The Newton-Schulz iteration they use approximates setting all singular values of the matrix to 1. That computes the nearest orthogonal matrix under the Frobenius norm.
[ Parent | on HN ]
Re:Matrix Orthogonalization Improves Memory in Recurrent Models by hasley on Wednesday July 01, 2026 @12:51PM (#48749773)
Interesting, thanks!
[ Parent | on HN ]
Re:Matrix Orthogonalization Improves Memory in Recurrent Models by CamperBob2 on Wednesday July 01, 2026 @01:32PM (#48750422)
3D graphics and kinematics people dodge the need for periodic orthonormalization by using quaternions. When they need a rotation matrix, they create it on demand rather than having to maintain it incrementally.

I wonder if there's a similar shortcut representation that we will eventually realize we should be using for ML. I suppose if there is one, it won't have native GPU support, so no one will bother looking for it.

[ Parent | on HN ]