Matrix Calculus for Deep Learning

Re:Matrix Calculus for Deep Learning by jph00 on Tuesday January 30, 2018 @01:48PM (#16267807)

I don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me.

The only tensor notation I've been happy with is that used in J (http://www.jsoftware.com), which is simple, flexible, and concise.

There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...

[ Parent | on HN ]

Re:Matrix Calculus for Deep Learning by jules on Tuesday January 30, 2018 @06:24PM (#16270049)

The notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply:

   A_ij = B_ik C_kj

Differentiating this with respect to variable l:

   ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj)

By writing out indices you can just use the rules for scalar derivatives.

[ Parent | on HN ]

Re:Matrix Calculus for Deep Learning by jph00 on Tuesday January 30, 2018 @08:05PM (#16270501)

Oh in that case I'm just confused - that's what I know as Einstein notation. Modern physics papers seemed to use much more complex notation, but I probably just misunderstood.

If we're talking about Einstein notation, then I'm a fan - `np.einsum()` is often a great way to create fast tensor computations with minimal code.

[ Parent | on HN ]