Whereas CS arrays tend to be 1 dimensional, and sometimes 2 dimensional, tensors can be as many dimensions as you need. A 256x256 photo with RGB channels would be stored as a [256 x 256 x 3] tensor/array. If you want to store a bunch of them? Add a dimension to store each image. Want rows and columns of images? Make the dimensions [width x height x channels x rows x columns].
Tools such as PyTorch's DataLoader can efficiently collate multiple inputs into a batch.
You can have arrays of as many dimensions as you need in many (most?) programming languages.
Is there some other difference between tensors and arrays?
Or is it just the math term for multidimensional array?
For the literal definition between a mathematical 'tensor' and a (e.g.) PyTorch 'tensor' (which, from reading the other comments here, are not the same thing), I leave that to other commenters. I'm just someone learning PyTorch and sharing my knowledge.
The practical engineering difference between a PyTorch tensor and Python array is that PyTorch comes with tons of math & utility functions that support n-dimensions and are differentiable, whereas the builtin Python arrays and math utilities don't do much, if any, of that.
A matrix is the coefficient list of a linear transformation, but it is not the transformation. The derivative operator is a linear transformation. It's representation as a matrix is just coordinates. But the operator is clearly more than that.
A tensor can be written down and you can compute with the components, but the tensor itself is the basis independent transformation. Not a list of components.
Understanding the inertia tensor on classical mechanics or the stress tensor may illustrate where tensors come from, and I understand that GR also makes use of a lot of tensor calculus that came to be as mathematics developed manipulating and talking about tensors. I have a kinda firm grasp on some very rudimentary tensor calculus from trying to leanr GR, and a pretty solid grasp on classical mechanics. I've had hour long conversations with deep learning popes and thought leaders in varying states of mind and after that my understanding is that they use the word tensor in an overreaching fashion as like you could call a solar panel a nuclear fission reactor power source. This thought leaders include people with books and 1M+ view Youtube videos on the subject that use the word tensor and I'm not saying their names because they off-the-record admitted that it's a poor choice of term but it harms the ego of many Google engineers to publicly admit that.
E.g., a type that holds "a 3x2 tensor OR a 4x6x2x5 tensor".
Rust to the rescue, as it always is for safety. (I don't know if this directly translates for tensors, I used it for manipulating point clouds and iirc it allows for arbitrary dimension containers)
Just a word of caution for anyone interested in using it.
Things like windowing, slicing, views are all “supported” but have many limitations and are non intuitive. In particular, I needed an “axis shift” fn and I ended up spending hours scouring docs just to find a way to shift along an axis. When I mean by shift is a roll in numpy, ala: https://github.com/rust-ndarray/ndarray/issues/281
Answer: you can’t with this crate. I implemented a dynamic n-dim solution myself but it uses views of integer indices that get copied to a new array, which have indexes to another flattened array in order to avoid duplication of possibly massive amounts of n-dimensional data; using the crate alone, copying all the array data would be unavoidable.
Ultimately I’ve had to make my own axis shifting and windowing mechanisms. But the crate is still a useful lib and continuing effort.
While I don’t mind getting into the weeds, these kinds of side efforts can really impact context focus so it’s just something to be aware of.
But really, the whitepaper should include this sort of detail, and you should be following the white paper.
In a context like this, we use tensor because it allows for any number of dimensions (while vector/ array is only one, matrix is two). When you get into ML libraries, both popular packages PyTorch and TensorFlow use the "tensor" terminology.
It's a good point. Hope it's clearer for devs with "array" terminology.
"Vector" implies one dimension and "matrix" strongly implies two. But an array can have any number of dimensions, so "array" is the best word.
We don't need the word "tensor"; when the context is programming, "tensor" is only confusing and doesn't really add any useful meaning.
And really, the context is math, not programming. The programming side of DL is approximately trivial, the interesting bits are all represented as mathematical operations on a set of tensors.
In programming too, multiplying two arrays of arbitrary types is often undefined, and many languages allow their programmers to specify what they want to happen when arrays of specific types are multiplied (or may have built-in defined behavior for certain specific types of arrays).
I'd love to learn if there are any actual differences between tensors and arrays commonly used in programming, but so far it doesn't sound like it from reading through this HN thread.
If I say "multiply this 3x3 array by this 3x3 array", then it's an ambiguous operation. There is not a canonical multiplication for arrays. One ecosystem might treat it as a matrix-matrix multiplication, another might do an element-wise multiplication, others might just throw an error. None is more correct than another, because array just implies the structure of the data.
Tensor (disambiguation)
https://en.wikipedia.org/wiki/Tensor_(disambiguation)
has an entry
Tensor type (computing)
which is just a redirection to
Array (data type)
https://en.wikipedia.org/wiki/Array_(data_type)
Doesn’t seem to support the relevance of “tensor” as something distinct from “array”.
(A mathematician might object to this on several grounds, such as that vectors/matrices/tensors are geometric objects which need not be expressed numerically as coordinates in any coordinate system)
Quote: "A tensor is a container which can house data in N dimensions. Often and erroneously used interchangeably with the matrix (which is specifically a 2-dimensional tensor), tensors are generalizations of matrices to N-dimensional space."
But simply put, a 'tensor' is a 3+-dimensional array of floats. Or a stack of matrices.
"To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it."
Then, you can visualize orthogonal on dimensions higher than 3 by throwing wires between equivalent points in two boxes.
Of course the problem with that is that you're still thinking three-dimensionally because "on top of" is a description of a 2D or 3D relationship.
It's like painting a house in isometric style. You draw each floor above the one below it, and put a "perpendicular" slanted axis to represent the third dimension in 2D.
You could put a 'slanted' fourth axis in a 3D origin of coordinates, but I find it easier to think of the lower corner of each box as the origin of coordinates displaced along that fourth axis.
Even with that the same problem occurs: "slanted" is a 3D relationship, so at best it's an analogy.
It's like "translating" the color red to some shade of gray to a person who can only see in black and white.
Or trying to describe through written words what music is like to someone who can't hear.
These are all analogies which might allow us to reason and get interesting/useful results when we use such analogies... but we should not under the illusion that we really know what those things for which we have no senses are like.
Also, there are probably all sorts of interesting/useful connections, relationships or conclusions that a being who really could sense 4D objects would find obvious or easy to make that we may never arrive at because our way of thinking of them is so limited in comparison.
That's not to mention trying to think about even higher dimensions or various other mathematical constructions that have no obvious translation to things in our ordinary experience we can relate to.
Yeah, but it's the same analogy you use to represent 3D figures on a 2D plane, where slanted is a 2D relationship; and we are very well versed on it and know how it works as a projection of a 3D reality.
> These are all analogies which might allow us to reason and get interesting/useful results when we use such analogies... but we should not under the illusion that we really know what those things for which we have no senses are like.
Is this some reflections about Plato's cave? You don't need much insight to understand the basis of moving over an extra dimension - if you can understand what it's like to move over parallel planes from a 2D view, you can understand the same by moving over superposed volumes from a 3D view. A hyper-sphere can be visualized as a linear collection of increasingly small spheres glued together, in the same way that you can view a sphere as a linear collection of increasingly small circles over an axis orthogonal to its center.
> Also, there are probably all sorts of interesting/useful connections, relationships or conclusions that a being who really could sense 4D objects would find obvious or easy to make that we may never arrive at because our way of thinking of them is so limited in comparison.
Yeah, and some people are really bad at 3D visualization so they have no hope to arrive at conclusions that are trivial to an architect or engineer. That doesn't prevent higher-dimension visualization techniques from being useful for the insights that it can provide, even if they can't give omniscience. For all the other insights, we have formal methods and logic reasoning, which is how 4D space was originated to begin with.
This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explain what that means” situation.
This is probably why I admire a whole bunch of engineering YouTubers and other engineering “PR” people for their brilliance at making complex stuff seem very very simple.