News for hackers. Stuff that matters.
Re:Tensor Considered Harmful Part 2 by amelius on Monday January 14, 2019 @03:22AM (#18901528)
By the way, why don't we just call them "arrays"?

Perhaps I'm missing something, but (practical) deep learning seems to have little tensor algebra going on, just operations on slices of arrays.

Re:Tensor Considered Harmful Part 2 by dogma1138 on Monday January 14, 2019 @03:39AM (#18901594)
I think the reasoning behind the nameing convention is that tensors in DL are just an abstraction of multi-dimensional arrays and that the operations you do on multiple arrays such as sum or multiply are algebraic tensor operations due to their multi linearity.

It could also be dated to earlier work on computer vision where tensors were used to store a bunch of spatial coordinates so it might be that back then it was even more aligned with their mathematical definition.

That said the grief doesn’t seem to be with the naming convention but rather with how tensor classes are designed in common DL frameworks.

http://nlp.seas.harvard.edu/NamedTensor

Re:Tensor Considered Harmful Part 2 by jjtheblunt on Monday January 14, 2019 @02:25PM (#18905624)
Mathematica has used the term tensor for multidimensional arrays since 1988. Sergei Brin worked at Wolfram Research for a summer internship or something back before Google, so would have been exposed to that nomenclature, as many in Urbana-Champaign were.
Re:Tensor Considered Harmful Part 2 by nabla9 on Monday January 14, 2019 @07:16AM (#18902452)
Mathematically tensors are multilinear maps, not data structures. Typically tensors in deep learning represent multilinear mappings forwards and backwards. Scalars, vectors and matrices can also be tensors when they represent mapping.

If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.

Re:Tensor Considered Harmful Part 2 by thanatropism on Monday January 14, 2019 @09:22AM (#18903102)
We've got a winner. Tensors "do" something; they're functions.

In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.

I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.

Re:Tensor Considered Harmful Part 2 by joshuamorton on Monday January 14, 2019 @12:47PM (#18904766)
>We've got a winner. Tensors "do" something; they're functions.

And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).

>neural networks as ODEs

You're a few months late :P [0]

[0]: https://arxiv.org/abs/1806.07366

Re:Tensor Considered Harmful Part 2 by twtw on Monday January 14, 2019 @10:47AM (#18903824)
Is it accurate to say that rank-n tensor is to n-D array of numbers as linear transformation is to matrix?
Re:Tensor Considered Harmful Part 2 by krastanov on Monday January 14, 2019 @11:16AM (#18904061)
For a mathematician or a physicist, yes. But the word ended up having new, confusing meanings.
Re:Tensor Considered Harmful Part 2 by nabla9 on Tuesday January 15, 2019 @08:16AM (#18910892)
It's the same for ML perspective as well.
Re:Tensor Considered Harmful Part 2 by Koshkin on Monday January 14, 2019 @07:14PM (#18907837)
> they're functions

Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.

Re:Tensor Considered Harmful Part 2 by thanatropism on Monday January 14, 2019 @10:09PM (#18908589)
Ehhh.

Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...

You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.

Re:Tensor Considered Harmful Part 2 by amelius on Tuesday January 15, 2019 @05:17AM (#18910090)
> Mathematically tensors are multilinear maps, not data structures.

Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.

Re:Tensor Considered Harmful Part 2 by contravariant on Monday January 14, 2019 @03:58AM (#18901661)
To the best of my knowledge the operations in deep learning are very much tensor operations. In most cases they seem to boil down to matrix multiplicaiton on one of the indices, but it's still kind of useful to use tensor notation (especially using the Einstein summation convention) since IMHO T_abc M_ax = S_xbc is easier to read than the corresponding sum, or the corresponding pseudocode.

Not sure if the fact that tensors have slightly more structure than mere arrays w.r.t. e.g. a change of basis is that useful, but it could be useful in proofs, especially if a FFT is involved somewhere.

Re:Tensor Considered Harmful Part 2 by mcabbott on Monday January 14, 2019 @06:43AM (#18902304)
Some operations are quite tensor-ish. For instance the re-shaping which einops writes like this

    Rearrange('b c h w -> b (c h w)')
is literally a tensor product of the vector spaces indexed by c,h,w. The same space that np.kron maps into (possibly modulo the order of indices) rather than np.hstack (of row vectors) which maps to the direct sum, ⊕ not ⊗.
Re:Tensor Considered Harmful Part 2 by imh on Monday January 14, 2019 @12:47PM (#18904759)
Once you put them into a particular coordinate basis, I'm pretty sure tensor algebra is just operation on slices of arrays. The magic happens when you change bases.

Its kinda like how a matrix is "just" a rectangular 2D array of numbers in a particular basis.

Re:Tensor Considered Harmful Part 2 by buboard on Monday January 14, 2019 @03:57AM (#18901655)
probably to avoid confusion with numpy arrays. and to irritate physicists
Re:Tensor Considered Harmful Part 2 by argonaut on Monday January 14, 2019 @04:09AM (#18901713)
Except they literally are numpy arrays (but on GPU).
Re:Tensor Considered Harmful Part 2 by singhrac on Monday January 14, 2019 @05:21AM (#18901992)
No, you're forgetting tape-based operation tracking (requires_grad=True).
Re:Tensor Considered Harmful Part 2 by vx100 on Monday January 14, 2019 @07:16AM (#18902451)
Yes to irritating physicists, no to avoiding confusion with numpy arrays. Pytorch README:

- "Tensor computation (like NumPy) with strong GPU acceleration"

- "If you use NumPy, then you have used Tensors (a.k.a ndarray)."

Pure marketing drivel.

Re:Tensor Considered Harmful Part 2 by buboard on Monday January 14, 2019 @07:26AM (#18902483)
i mean, they have different names when used in pytorch code, tensor vs ndarray.