Matrix Orthogonalization Improves Memory in Recurrent Models

Re:Matrix Orthogonalization Improves Memory in Recurrent Models by bee_rider on Wednesday July 01, 2026 @10:44AM (#48747692)

I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can always do that, but I’m sure that’s been tried already.

[ Parent | on HN ]

Re:Matrix Orthogonalization Improves Memory in Recurrent Models by phkahler on Thursday July 02, 2026 @09:23AM (#48761187)

But orthogonal matrices are square.

[ Parent | on HN ]

Re:Matrix Orthogonalization Improves Memory in Recurrent Models by bee_rider on Thursday July 02, 2026 @01:50PM (#48765013)

Ah, good point. I assumed they were coming up with some orthogonal basis vectors for a (potentially) non-square matrix. Edit: actually I’m not sure, this Newton-Schulz process seems to work for non-square matrices as well. Generally I see “orthogonalization” refer to the process of coming up with those orthogonal basis vectors but it could be a domain-specific lingo thing.

[ Parent | on HN ]