News for hackers. Stuff that matters.
Re:Vectorization: Introduction by alfalfasprout on Thursday June 01, 2023 @05:23PM (#36157532)
Tensor ops are usually implemented using the principles of vectorizing code.

After all most tensors are implemented as 1D arrays with strides for each dimension with the innermost dimension always contiguous.

Re:Vectorization: Introduction by sva_ on Thursday June 01, 2023 @05:36PM (#36157697)
I think the bigger difficulty lies in how to store and process this stuff on a GPU/other accelerator.

If the OP cares about implementation details about how an API like PyTorch is made, I think the MiniTorch 'book' is a pretty good intro:

https://minitorch.github.io/

Re:Vectorization: Introduction by lying4fun on Thursday June 01, 2023 @07:28PM (#36158862)
The MiniTorch book seems amazing, thanks
Re:Vectorization: Introduction by richardw on Thursday June 01, 2023 @05:28PM (#36157599)
ChatGPT interpretation:

Sure! Let me break it down for you:

When we talk about "tensor ops," we're referring to operations performed on tensors, which are multidimensional arrays of numbers commonly used in mathematics and computer science.

Now, these tensor operations are typically implemented using a technique called "vectorizing code." In simple terms, vectorizing code means performing operations on entire arrays of data instead of looping through each element one by one. It's like doing multiple calculations at once, which can be more efficient and faster.

Tensors are usually represented as 1D arrays, meaning all the elements are arranged in a single line. Each dimension of the tensor has a concept called "stride," which represents how many elements we need to skip to move to the next element in that dimension. This helps us efficiently access and manipulate the data in the tensor.

Additionally, the innermost dimension of a tensor is always "contiguous," which means the elements are stored sequentially without any gaps. This arrangement also aids in efficient processing of the tensor data.

So, in summary, tensor ops involve performing operations on multidimensional arrays, and we use vectorized code to do these operations more efficiently. Tensors are represented as 1D arrays with strides for each dimension, and the innermost dimension is always stored sequentially without any gaps.

Re:Vectorization: Introduction by riku_iki on Thursday June 01, 2023 @05:57PM (#36157951)
> the principles of vectorizing code. > 1D arrays

I think accelerators mostly do computations over small matrices and not vectors, so it is "matricized" code, but I am not an expert in this area.