Why GEMM is at the heart of deep learning

Re:Why GEMM is at the heart of deep learning by gcr on Monday April 20, 2015 @09:39PM (#9411841)

Oops! The "A x B = C" figure is exactly backwards. A matrix with shape (r, k) times a matrix with shape (k, c) should return a matrix with shape (r,c). The figure incorrectly claims that a matrix (k,m) times a matrix with shape (n,k) returns a matrix with shape (n,m), which is only true if all of these numbers are equal (eg. square matrices).

Kind of an embarrassing typo for an article that is essentially about matrix multiplication.

Here is a corrected (transposed) version of that figure: http://i.imgur.com/Bo6CJCv.png

Edit: It looks like every figure in the article suffers from this problem. I know FORTRAN matrices store each column contiguously in memory, but whether a matrix is accessed like * (ptr + r * ncols + c) or * (ptr + c * nrows + r) is (in my opinion) a low-level detail and shouldn't affect our thinking about how multiplication works.

[ Parent | on HN ]

Re:Why GEMM is at the heart of deep learning by simonster on Monday April 20, 2015 @11:34PM (#9412146)

Either I'm missing something here or the article is correct. Generally, the first dimension of a matrix corresponds to the rows and the second dimension to the columns. So in the original figure, A is m x k, B is k x n, and C is m x n, and this seems right. See http://en.wikipedia.org/wiki/Matrix_%28mathematics%29#Size

[ Parent | on HN ]

Re:Why GEMM is at the heart of deep learning by dplarson on Tuesday April 21, 2015 @01:36AM (#9412437)

I think the author revised the figure(s) between the time of the parent comment (by gcr) and your comment. At least, the A * B = C figure's filename seems to imply a revision [1].

EDIT: yep, the figures were revised. Compare the corrected version [1] vs the original [2].

[1] https://petewarden.files.wordpress.com/2015/04/gemm_correcte...

[2] https://petewarden.files.wordpress.com/2015/04/gemm.png

EDIT 2: I completely missed that the author put a notice (about having revised the figures) at the bottom of the post.

[ Parent | on HN ]

Re:Why GEMM is at the heart of deep learning by petewarden on Tuesday April 21, 2015 @02:26AM (#9412542)

Yes, that was my screwup, sorry for any confusion! Despite multiple linear algebra courses, working in 3D graphics for a decade, I haven't internalized the basics of matrix notation. It doesn't help that my usual sandbox (Eigen) is column major by default, which is the wrong in-memory order for my raster-image trained brain to visualize. Funnily enough, I find tensor notation a bit easier, despite being less familiar.

[ Parent | on HN ]

Re:Why GEMM is at the heart of deep learning by dplarson on Saturday April 25, 2015 @12:08AM (#9437155)

No worries. Also, I found the article very interesting and informative. Thanks!

[ Parent | on HN ]