Transformer learning explained: Coinductive guide to inductive transformer heads

Re:Transformer learning explained: Coinductive guide to indu... by ftxbro on Tuesday February 28, 2023 @12:48PM (#34972387)

I don't know anything about algebraic geometry or category theory or haskell, but I know some stuff about the normal boring kind of linear algebra, and in some places the author relies on statements like "since the eigenvalues are real and positive we can conclude that it is a symmetric positive definite matrix" which I know are just wrong (it wouldn't have to be symmetric). This is particularly bad because the rest of the argument (for example in 6.2.2) explains why it's so cool that we've shown it's symmetric (we haven't).

[ Parent | on HN ]

Re:Transformer learning explained: Coinductive guide to indu... by adamnemecek on Tuesday February 28, 2023 @01:37PM (#34973025)

You are right, I will rectify that. The statement still holds I just need to provide a justification.

[ Parent | on HN ]

Re:Transformer learning explained: Coinductive guide to indu... by adamnemecek on Tuesday February 28, 2023 @08:03PM (#34977358)

Just FYI when you lookup visualizations of the attention matrices, they tend to be somewhat symmetric.

[ Parent | on HN ]

Re:Transformer learning explained: Coinductive guide to indu... by east2west on Wednesday March 01, 2023 @03:33AM (#34980060)

I think that sentence is confusing but mathematical conclusion still stands. For quadratic forms, we are only concerned with symmetric matrices, or the symmetric part of any real matrix, because we can always replace any square matrix in a quadratic form, x^T A x, by x^T (A + A^T)/2 x, the symmetric part of the matrix, and the quadratic form still retains the original value. Then, for symmetric matrices, their eigenvalues being positive implies that the matrices are positive definite. Symmetric square matrices have real eigenvalues; one can prove this by taking the complement of quadratic forms.

[ Parent | on HN ]

Re:Transformer learning explained: Coinductive guide to indu... by matheist on Tuesday February 28, 2023 @12:59PM (#34972529)

For example, [[1, 1], [0, 1]] is a non-symmetric matrix with positive real eigenvalues.

[ Parent | on HN ]