Re:A 100x speedup with unsafe Python
by SailorJerry on Tuesday May 07, 2024 @01:04AM
(#40282510)
I think the reason I prefer columns is I do the mental expansion into large bracketed expressions. If x is a row and kept inline, then the expansion gets really wide. To keep it compact and have the symbols oriented the same as their expansion, you'd have to put the x above A and that's just silly.
Well, there used to be a tradition in universal algebra (and category theory) of putting functions/operators at the right side of the arguments but it seems to have ultimately died out. And "y = xA" is the standard notation in the linear coding theory, even to this day: message vectors are bit strings, not bit columns.
I think there’s also some other pretty big field that tends to put the vector on the left… statistics? Economics? For some reason I couldn’t find any links, so maybe I just got this from, like, one random statistician. If only they’d told me about sample sizes.
Your reasoning starts out nicely with y=Ax, but you get the wrong conclusion. The layout of x and y are not affected at all by row or column order, they are just consecutive numbers in either case. So you have to look at the layout of A.
For the row-major layout the multiplication Ax ends up being more cache-efficient than for the column major layout, as for calculating each component of y you need to scan x and a row of A.
I’m pretty sure BLAS will tile out the matrix multiplication internally anyway, so it doesn’t matter. At least for matmat would, is matvec special?
However matmat is done, row-major vs column-major for both matrices shouldn't make a difference (for square matrices at least).
I don't know if tiling is done for matvec. I don't think it makes sense there, but I didn't think about it too hard.
Re:A 100x speedup with unsafe Python
by montebicyclelo on Tuesday May 07, 2024 @07:55AM
(#40284454)
y = x @ W + b
Is how you'd write it in NumPy and most (/all?) deep learning libraries, with x.shape=[batch, ndim]
I'd personally prefer that to be the convention, for consistency.
So, x and y must be columns. So, it is nice if our language has column-order as the default, that way a vector is also just a good old 1d array.
If we did y=xA, x and y would be rows, the actual math would be the same but… I dunno, isn’t it just hideous? Ax is like a sentence, it should start with an upper case!
It also fits well with how we tend to learn math. First we learn about functions, f(x). Then we learn that a function can be an operator. And a matrix can represent a linear operator. So, we tend to have a long history of stuff sitting to the left of x getting ready to transform it.