News for hackers. Stuff that matters.
Re:Vectorization: Introduction by nologic01 on Thursday June 01, 2023 @05:20PM (#36157503)
We need also something like Tensorization:Introduction
Re:Vectorization: Introduction by alfalfasprout on Thursday June 01, 2023 @05:23PM (#36157532)
Tensor ops are usually implemented using the principles of vectorizing code.

After all most tensors are implemented as 1D arrays with strides for each dimension with the innermost dimension always contiguous.

Re:Vectorization: Introduction by sva_ on Thursday June 01, 2023 @05:36PM (#36157697)
I think the bigger difficulty lies in how to store and process this stuff on a GPU/other accelerator.

If the OP cares about implementation details about how an API like PyTorch is made, I think the MiniTorch 'book' is a pretty good intro:

https://minitorch.github.io/

Re:Vectorization: Introduction by lying4fun on Thursday June 01, 2023 @07:28PM (#36158862)
The MiniTorch book seems amazing, thanks
Re:Vectorization: Introduction by richardw on Thursday June 01, 2023 @05:28PM (#36157599)
ChatGPT interpretation:

Sure! Let me break it down for you:

When we talk about "tensor ops," we're referring to operations performed on tensors, which are multidimensional arrays of numbers commonly used in mathematics and computer science.

Now, these tensor operations are typically implemented using a technique called "vectorizing code." In simple terms, vectorizing code means performing operations on entire arrays of data instead of looping through each element one by one. It's like doing multiple calculations at once, which can be more efficient and faster.

Tensors are usually represented as 1D arrays, meaning all the elements are arranged in a single line. Each dimension of the tensor has a concept called "stride," which represents how many elements we need to skip to move to the next element in that dimension. This helps us efficiently access and manipulate the data in the tensor.

Additionally, the innermost dimension of a tensor is always "contiguous," which means the elements are stored sequentially without any gaps. This arrangement also aids in efficient processing of the tensor data.

So, in summary, tensor ops involve performing operations on multidimensional arrays, and we use vectorized code to do these operations more efficiently. Tensors are represented as 1D arrays with strides for each dimension, and the innermost dimension is always stored sequentially without any gaps.

Re:Vectorization: Introduction by riku_iki on Thursday June 01, 2023 @05:57PM (#36157951)
> the principles of vectorizing code. > 1D arrays

I think accelerators mostly do computations over small matrices and not vectors, so it is "matricized" code, but I am not an expert in this area.

Re:Vectorization: Introduction by corsix on Thursday June 01, 2023 @05:57PM (#36157945)
From a hardware perspective, vector instructions operate on small 1D vectors, whereas tensor instructions operate on small 2D matrices. I say “instructions”, but it’s really only matrix multiply or matrix multiply and accumulate - most other instructions are fine staying as 1D.
Re:Vectorization: Introduction by nologic01 on Thursday June 01, 2023 @06:07PM (#36158068)
If there is matrix multiply at hardware level its fair to have another name than vectorization. For example the dimensions and partitioning of large matrices to fit would be specific to that design and very different from rolling things out on 1D arrays
Re:Vectorization: Introduction by bee_rider on Friday June 02, 2023 @12:43AM (#36161140)
Maybe they should call it a Matrix Multiplication Xcelerator just to mess with people.
Re:Vectorization: Introduction by amelius on Thursday June 01, 2023 @06:05PM (#36158050)
We really need to name tensors differently. "Array" would be a more fitting name.

Tensors are really different mathematical objects with a far more rich structure than those used in the Deep Learning context.

https://en.wikipedia.org/wiki/Tensor

Re:Vectorization: Introduction by sva_ on Thursday June 01, 2023 @06:28PM (#36158297)
Names can have different meanings based on context, even within pure mathematics. I think a mathematician would have little trouble discerning which kind of tensor is meant, as this sort of reusing existing terms is pretty common.

Just calling it an array might be underselling it, at this point. Perhaps a tensor in the context of CompSci is just a particular type of array with certain expected properties.

Re:Vectorization: Introduction by rsfern on Thursday June 01, 2023 @09:29PM (#36159756)
> Just calling it an array might be underselling it, at this point

I like the NDArray terminology used by numpy. I think it gets the point across more clearly than “tensor”, and conceptually they’re pretty much equivalent

Re:Vectorization: Introduction by itishappy on Thursday June 01, 2023 @06:11PM (#36158105)
Re:Vectorization: Introduction by antognini on Friday June 02, 2023 @01:05AM (#36161244)
As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. (They are one-forms.) But in the ML context it's a distinction without a difference, unless you're in a really specialized corner of the field like information geometry.
Re:Vectorization: Introduction by ReleaseCandidat on Friday June 02, 2023 @04:09AM (#36162481)
> Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors.

In linear algebra you should learn this as soon as you hear about it. Is the english Wikipedia page correct in that you always use nabla for the gradient? https://en.wikipedia.org/wiki/Gradient#Generalizations

Because `grad f = ∇f` holds in scalar fields f only (not in vector or tensor fields (tensors that aren't scalars)).

Re:Vectorization: Introduction by antognini on Friday June 02, 2023 @01:07PM (#36167839)
Even if you are taking the gradient of a scalar field, the resulting object is not a vector. The difference is that if you apply some transformation to your coordinate system a gradient transforms differently from a normal vector.

In the ML context it generally doesn't matter because you don't do these transformations (and more generally there is no metric for your space). So you can just treat a gradient as an array of numbers. But in a physical context the distinction starts to matter, at least on a manifold with curvature.

The one place it does matter in ML (that I'm aware of) is information geometry, where you try to do a transformation from the normal coordinate space to a more "natural" coordinate system that is based on the Fisher information, and this introduces curvature into model.

Re:Vectorization: Introduction by opportune on Thursday June 01, 2023 @06:17PM (#36158170)
You could apply that argument to “isomorphic” functions, c++ “vectors” or compute “vectorizarion”, “integer” types… even though the semantics aren’t exactly the same between the computing and math terms, they’re related enough that they do help with understanding to those coming from a math background (which to be fair is probably less and less programmers over time).
Re:Vectorization: Introduction by myownpetard on Thursday June 01, 2023 @06:30PM (#36158313)
Isomorphic javascript is one of the most egregious.
Re:Vectorization: Introduction by nologic01 on Thursday June 01, 2023 @06:20PM (#36158214)
I think that naming battle is lost.

Would indeed be cool to have true tensor processing units :-)

PKD level of cool as you could argue those chips directly process spacetime chunks

Re:Vectorization: Introduction by sroussey on Thursday June 01, 2023 @07:35PM (#36158913)
Tensor processing units sounds far more cutting edge than number array processing units.
Re:Vectorization: Introduction by godelski on Thursday June 01, 2023 @06:32PM (#36158339)
This is mostly going to be the same. The main difference in tensorization is that your vectors are not contiguous. So you lose some speedup. How you typically optimize this is by flattening your tensor. Then your data is contiguious and all the same rules apply.

This is overly simplified though. Things get different when we start talking about {S,M}I{S,M}D (page addresses SIMD) or GPUs. Parallel computing is a whole other game, and CUDA takes it to the next level (lots of people can write CUDA kernels, not a lot of people can write GOOD CUDA kernels (I'm definitely not a pro, but know some)). Parallelism adds another level of complexity due to being able to access different memory locations simultaneously. If you think concurrent optimization is magic, parallel optimization is wizardry (I still believe this is true)

Re:Vectorization: Introduction by quickthrower2 on Thursday June 01, 2023 @07:50PM (#36159038)
PyTorch internals: http://blog.ezyang.com/2019/05/pytorch-internals/

A good basic thing to know is a tensor is a single dimensional array under the hood. At least as far as I know in my mental model - I have not yet looked at the code :-)

Therefore "converting" a 2 * 5 * 10 tensor into a 2 * 50 one is an immediate operation, it doesn't have to scan. Just change some metadata.

Re:Vectorization: Introduction by jabl on Friday June 02, 2023 @03:20AM (#36162109)
That's pretty much how every serious dense matrix (or more generally, multi-dimensional array) representation has been done since, well, forever. It's only C programmers writing "My First Matrix Program" that insist on allocating each row separately due to the lack of native multidimensional arrays in that language.
Re:Vectorization: Introduction by quickthrower2 on Friday June 02, 2023 @03:30AM (#36162181)
Yes! I was surprised how simple it was. Obvious once you know how. I never really considered multidimensional storage until looking into Machine Learning recently. My only brush with it is relational database tuning, but that is very different, as the number of "dimensions" are small and they are usually not homogenous.
Re:Vectorization: Introduction by ReleaseCandidat on Friday June 02, 2023 @04:21AM (#36162581)
Fortran did it:

    A 2-dimensional array A will, in the object program, be stored sequentially in
    the order Ai,i, A2,i Am l , A| i2 , A2f2 Am, 2> , Am,„. Thus
    it is stored “columnwise”, with the first of its subscripts varying most 
    rapidly, and the last varying least rapidly. The same is true of 3-dimensional arrays.
   1 -dimensional arrays are of course simply stored sequentially. All arrays are stored backwards in storage; i.e. the above sequence is in the order of decreasing absolute location.
https://ia800807.us.archive.org/0/items/history-of-fortran/I...