Re:Einsum Is All You Need – Einstein Summation in Deep Learn...
by mmmmpancakes on Sunday February 03, 2019 @06:43PM
(#19072943)
Gradients are vectors. You can certainly add them to other vectors.
I think what you mean to say is that you can not add vectors and covectors. The differential of a function is a 1-form. Pointwise, it is a covector. This data is the natural starting point from which to compute gradients.
Metrics can be used to convert covectors into vectors and forms into vector fields. The result is the thing you call the natural gradient. If you do this with the standard metric on R^n, then you get the vector field which you call the gradient.
I think what you mean to say is that you can not add vectors and covectors. The differential of a function is a 1-form. Pointwise, it is a covector. This data is the natural starting point from which to compute gradients.
Metrics can be used to convert covectors into vectors and forms into vector fields. The result is the thing you call the natural gradient. If you do this with the standard metric on R^n, then you get the vector field which you call the gradient.