The Matrix Calculus You Need for Deep Learning

Re:The Matrix Calculus You Need for Deep Learning by quanto on Monday July 31, 2023 @12:55PM (#36945465)

The article/webpage is a nice walk-through for the uninitiated. Half the challenge of doing matrix calculus is remembering the dimension of the object you are dealing with (scalar, vector, matrix, higher-dim tensor).

Ultimately, the point of using matrix calculus (or matrices in general) is not just concision of notation but also understanding that matrices are operators acting on members of some spaces, i.e. vectors. It is this higher level abstraction that makes matrices powerful.

For people who are familiar with the concepts but need a concise refresher, the Wikipedia page serves well:

https://en.wikipedia.org/wiki/Matrix_calculus

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by PartiallyTyped on Monday July 31, 2023 @01:22PM (#36945894)

Adding, these operators are also "polymorphic"; for matrix multiplication the only operations you need are (non commutative) multiplication and addition; thus you can use elements of any non-commutative ring, i.e. a set of elements with those two operations :D

Matrices themselves form non-commutative rings too; and based on this, you can think of a 4N x 4N matrix as a 4x4 matrix whose elements are NxN matrices [1] :D

[1] https://youtu.be/FX4C-JpTFgY?list=PL49CF3715CB9EF31D&t=1107

You already know whose lecture it is :D

I love math.. I should have become a mathematician ...

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by mrfox321 on Monday July 31, 2023 @01:37PM (#36946113)

Re [1]: it's fairly concrete to simply say that matrix multiplication can be performed block-wise.

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by PartiallyTyped on Monday July 31, 2023 @02:25PM (#36946777)

I don’t disagree; but that is just an example of MM. The gist is not that you can do block multiplication; but that you can define matrices over any valid ring, which includes other matrices - ie blocks.

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by mrfox321 on Monday July 31, 2023 @03:20PM (#36947501)

Yeah matrices are more abstract. I guess I am just pointing out that your concrete example of non-commutative rings (matrices of matrices) still needs a proof to demonstrate bijection between 4N x 4N (scalar) and 4 x 4 (N x N(scalar)).

Block MM demonstrates the equivalence.

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by tikhonj on Monday July 31, 2023 @08:13PM (#36950671)

You can even generalize linear algebra algorithms to closed semirings and have some really cool algorithms pop out, like finding the shortest path in graphs. There's a great paper called "Fun with Semirings" that goes into more details; unfortunately looks like the PDF isn't easily available online any more, but I found some slides[1] that seem to cover the same ideas well enough.

[1]: https://pdfs.semanticscholar.org/2e43/477e26a54b2d1a046c2140...

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by PartiallyTyped on Tuesday August 01, 2023 @02:39AM (#36953053)

This deserves its own HN post imho.

[ Parent | on HN ]

Re:The Matrix Calculus You Need for Deep Learning by PartiallyTyped on Tuesday August 01, 2023 @06:47AM (#36954495)

Okay I went over the slides and good lord this would have made my life easier not too long ago.

[ Parent | on HN ]