News for hackers. Stuff that matters.
Re:The Illustrated Stable Diffusion by Waterluvian on Tuesday October 04, 2022 @04:09PM (#33086005)
This helps. Thank you. Any advice on where to look to understand why the word tensor was used?
Re:The Illustrated Stable Diffusion by pvarangot on Tuesday October 04, 2022 @04:56PM (#33086723)
A Tensor is a mathematical object for symbolic manipulation of relationships between other objects that belong in conceptually similar universes or spaces. Literature on Deep Learning, like Goodfellow, call for the CS-minded reader to just assume it's a fancy word for a matrix of more than two dimensions. That makes matters confusing because mathematically you could have scalar or vectorial tensors. The classic mathematical definition puts more restrictions on the "shape" of the "matrix" by requiring certain properties so that it can create a symbolic language for a tensor calculus. Then you can study how the relationships change as variables on the universes or spaces change.

Understanding the inertia tensor on classical mechanics or the stress tensor may illustrate where tensors come from, and I understand that GR also makes use of a lot of tensor calculus that came to be as mathematics developed manipulating and talking about tensors. I have a kinda firm grasp on some very rudimentary tensor calculus from trying to leanr GR, and a pretty solid grasp on classical mechanics. I've had hour long conversations with deep learning popes and thought leaders in varying states of mind and after that my understanding is that they use the word tensor in an overreaching fashion as like you could call a solar panel a nuclear fission reactor power source. This thought leaders include people with books and 1M+ view Youtube videos on the subject that use the word tensor and I'm not saying their names because they off-the-record admitted that it's a poor choice of term but it harms the ego of many Google engineers to publicly admit that.

Re:The Illustrated Stable Diffusion by amelius on Tuesday October 04, 2022 @06:46PM (#33088167)
How are sum-types handled in deep learning?

E.g., a type that holds "a 3x2 tensor OR a 4x6x2x5 tensor".

Re:The Illustrated Stable Diffusion by nicd on Tuesday October 04, 2022 @07:52PM (#33088888)
I'm not aware of tensor shape type safety (in general) being widely used in deep learning, let alone sum-types. I believe Pytorch and TensorFlow lack support for tensor shape type checking (looks like there is a Pytorch issue open: https://github.com/pytorch/pytorch/issues/26889).
Re:The Illustrated Stable Diffusion by ntr-- on Tuesday October 04, 2022 @11:02PM (#33090601)
https://docs.rs/ndarray/0.15.6/ndarray/

Rust to the rescue, as it always is for safety. (I don't know if this directly translates for tensors, I used it for manipulating point clouds and iirc it allows for arbitrary dimension containers)

Re:The Illustrated Stable Diffusion by survirtual on Wednesday October 05, 2022 @04:50PM (#33101102)
I use this crate extensively and while I have gotten good results, it is not ergonomic and it is not equivalent to numpy ndarray. There are a lot of issues and work still to be done.

Just a word of caution for anyone interested in using it.

Things like windowing, slicing, views are all “supported” but have many limitations and are non intuitive. In particular, I needed an “axis shift” fn and I ended up spending hours scouring docs just to find a way to shift along an axis. When I mean by shift is a roll in numpy, ala: https://github.com/rust-ndarray/ndarray/issues/281

Answer: you can’t with this crate. I implemented a dynamic n-dim solution myself but it uses views of integer indices that get copied to a new array, which have indexes to another flattened array in order to avoid duplication of possibly massive amounts of n-dimensional data; using the crate alone, copying all the array data would be unavoidable.

Ultimately I’ve had to make my own axis shifting and windowing mechanisms. But the crate is still a useful lib and continuing effort.

While I don’t mind getting into the weeds, these kinds of side efforts can really impact context focus so it’s just something to be aware of.

Re:The Illustrated Stable Diffusion by gpderetta on Wednesday October 05, 2022 @04:12AM (#33092679)
How does this help with typechecking tensor types in python?
Re:The Illustrated Stable Diffusion by mxkopy on Wednesday October 05, 2022 @03:10AM (#33092228)
In my experience (CV) you typically want 4 dims - (height, width, channels, batch). In Julia you can do element-wise operations on two arrays if their lengths (i.e. size after flattening) are the same. Obviously dimension-wise that sort of thing won't work (what's the dot product of a 3-vector by a 6-vector? Undefined!)

But really, the whitepaper should include this sort of detail, and you should be following the white paper.

Re:The Illustrated Stable Diffusion by jayalammar on Tuesday October 04, 2022 @04:49PM (#33086607)
I updated the post to say "multi-dimensional array".

In a context like this, we use tensor because it allows for any number of dimensions (while vector/ array is only one, matrix is two). When you get into ML libraries, both popular packages PyTorch and TensorFlow use the "tensor" terminology.

It's a good point. Hope it's clearer for devs with "array" terminology.

Re:The Illustrated Stable Diffusion by zestyping on Tuesday October 04, 2022 @06:34PM (#33088050)
> we use tensor because it allows for any number of dimensions

"Vector" implies one dimension and "matrix" strongly implies two. But an array can have any number of dimensions, so "array" is the best word.

We don't need the word "tensor"; when the context is programming, "tensor" is only confusing and doesn't really add any useful meaning.

Re:The Illustrated Stable Diffusion by OkayPhysicist on Tuesday October 04, 2022 @07:15PM (#33088469)
Tensor does imply a set of operations that are expected. Multiplying two arrays together is an ambiguous operation; multiplying two tensors together is well-defined.

And really, the context is math, not programming. The programming side of DL is approximately trivial, the interesting bits are all represented as mathematical operations on a set of tensors.

Re:The Illustrated Stable Diffusion by pmoriarty on Wednesday October 05, 2022 @10:38AM (#33096133)
"Tensor does imply a set of operations that are expected. Multiplying two arrays together is an ambiguous operation; multiplying two tensors together is well-defined."

In programming too, multiplying two arrays of arbitrary types is often undefined, and many languages allow their programmers to specify what they want to happen when arrays of specific types are multiplied (or may have built-in defined behavior for certain specific types of arrays).

I'd love to learn if there are any actual differences between tensors and arrays commonly used in programming, but so far it doesn't sound like it from reading through this HN thread.

Re:The Illustrated Stable Diffusion by OkayPhysicist on Wednesday October 05, 2022 @06:42PM (#33102144)
I think you misunderstood what I meant. If I say "multiply this 3x3 matrix by this 3x1 vector", then I know know to expect the particular form of multiplication that multiplies each column of the matrix by it's associated value in the vector, then adds them up to produce a 3x1 vector. Likewise, multiplying a 3x3x3 tensor by a 3x1x1 tensor will produce a 3x3x1 result.

If I say "multiply this 3x3 array by this 3x3 array", then it's an ambiguous operation. There is not a canonical multiplication for arrays. One ecosystem might treat it as a matrix-matrix multiplication, another might do an element-wise multiplication, others might just throw an error. None is more correct than another, because array just implies the structure of the data.

Re:The Illustrated Stable Diffusion by kgwgk on Wednesday October 05, 2022 @03:17AM (#33092279)
Interestingly the Wikipedia page

Tensor (disambiguation)

https://en.wikipedia.org/wiki/Tensor_(disambiguation)

has an entry

Tensor type (computing)

which is just a redirection to

Array (data type)

https://en.wikipedia.org/wiki/Array_(data_type)

Doesn’t seem to support the relevance of “tensor” as something distinct from “array”.

Re:The Illustrated Stable Diffusion by leereeves on Tuesday October 04, 2022 @09:01PM (#33089604)
It's especially confusing as math, because the standard meaning of "tensor" in math includes a lot of baggage that's not used in deep learning (related to physics and differential geometry).
Re:The Illustrated Stable Diffusion by zestyping on Wednesday October 05, 2022 @05:05PM (#33101239)
Exactly.
Re:The Illustrated Stable Diffusion by FeepingCreature on Wednesday October 05, 2022 @12:46AM (#33091364)
"Array" implies one dimension to me. Maybe n-array?
Re:The Illustrated Stable Diffusion by kgwgk on Wednesday October 05, 2022 @02:43AM (#33092082)
Array may imply 1d to you, or 2d to others, but it is more general that that: https://en.wikipedia.org/wiki/Array_(data_type)
Re:The Illustrated Stable Diffusion by auggierose on Wednesday October 05, 2022 @03:57AM (#33092574)
I think tensor is much shorter than "multi-dimensional array", and it is OK to use it. You could just mention that in the context of DL tensor is just a shorter way to say "multi-dimensional array", and everyone should be happy. On top of that, if they read another text about DL that uses the word tensor, there is no need to be scared anymore.
Re:The Illustrated Stable Diffusion by kgwgk on Wednesday October 05, 2022 @02:55AM (#33092152)
It’s a cool name from the good ol’ “multi-dimensional arrays with a uniform type”

https://www.tensorflow.org/guide/tensor

Re:The Illustrated Stable Diffusion by sva_ on Tuesday October 04, 2022 @04:44PM (#33086529)
It should be noted that it doesn't have all to much to do with the rigorous mathematical definition of a Tensor.
Re:The Illustrated Stable Diffusion by jamessb on Tuesday October 04, 2022 @05:18PM (#33087064)
Very loosely, a number/vector/matrix/tensor can be considered to be objects where specifying the values of 0/1/2/3 indexes will give a number.

(A mathematician might object to this on several grounds, such as that vectors/matrices/tensors are geometric objects which need not be expressed numerically as coordinates in any coordinate system)

Re:The Illustrated Stable Diffusion by amelius on Tuesday October 04, 2022 @06:43PM (#33088135)
A tensor can take any number of dimensions (not just 3).
Re:The Illustrated Stable Diffusion by avereveard on Tuesday October 04, 2022 @04:35PM (#33086416)
To reduce it a little, matrix holds numbers, tensor holds whatever. Numbers. Vectors. Operations.
Re:The Illustrated Stable Diffusion by yreg on Tuesday October 04, 2022 @06:30PM (#33087998)
So it is math lingo for array.
Re:The Illustrated Stable Diffusion by squeaky-clean on Wednesday October 05, 2022 @04:53AM (#33092948)
In math tensor has a more specific meaning than just an array, but in machine learning lingo, tensor just means an array or multidimensional array. That base is true in math as well, but it can also be a single value (rank zero tensor, aka a scalar. Sometimes ML will call this a tensor as well) and has to obey certain rules about linearity for some operations applied to it.