Re:Matrix Calculus for Deep Learning
by jules on Tuesday January 30, 2018 @01:26PM
(#16267611)
What do you think about the index notation physicists use for tensor calculus?
[ Parent
| on HN ]
I don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me.
The only tensor notation I've been happy with is that used in J (http://www.jsoftware.com), which is simple, flexible, and concise.
There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...
The notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply:
A_ij = B_ik C_kj
Differentiating this with respect to variable l: ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj)
By writing out indices you can just use the rules for scalar derivatives.Oh in that case I'm just confused - that's what I know as Einstein notation. Modern physics papers seemed to use much more complex notation, but I probably just misunderstood.
If we're talking about Einstein notation, then I'm a fan - `np.einsum()` is often a great way to create fast tensor computations with minimal code.
index notation isn't useful for calculation it's useful for algebra i.e. if you want closed form solution to tensor equations. my distinct impression is that in ML no one cares about that because eventually everything gets a numerical treatment.