Numpy was originally designed for mathematicians and scientists, and in those domains convention often lines up better with column major order. For example, vectors are a single column, and the column index comes before the row index in many notations. So using column major order meant familar formulas and algorithms (including translating from fortran code which is column major, or matlab for that matter) are easy to translate to numpy code.

Also, numpy is built on some fortran libraries, like BLAS and LAPACK that assume a column major order. If it took row major input, it would need to transpose matrices in some cases change the order, or rewrite those libraries to use row major form.

Copying is usually not necessary. Often times you can swap data and/or shape arguments and get a transposed result out. While it is true that Fortran BLAS only supports col-major, CBLAS supports both row- and col-major. Internally, all the libraries I worked on use col-major, but that is just a convention.

Probably because Fortran stores matrices and other multidimensional arrays in column order. Traditionally most numerical computation software was written in Fortran and numpy calls those under the hood. Storing in row order would have meant copying the data to column major order and back for any call to Fortran.

NumPy can _store_ arrays in either row-major (C-style) or column-major (Fortran-style) order. Row-major is the default. Index order is always row, column.

It is possible that my brain had been damaged by convention, but it looks much tidier to have your vector multiplicand on the right of the matrix when doing a matrix vector multiplication y=Ax, where A is the matrix and x is the vector, y is the result.

So, x and y must be columns. So, it is nice if our language has column-order as the default, that way a vector is also just a good old 1d array.

If we did y=xA, x and y would be rows, the actual math would be the same but… I dunno, isn’t it just hideous? Ax is like a sentence, it should start with an upper case!

It also fits well with how we tend to learn math. First we learn about functions, f(x). Then we learn that a function can be an operator. And a matrix can represent a linear operator. So, we tend to have a long history of stuff sitting to the left of x getting ready to transform it.

I think the reason I prefer columns is I do the mental expansion into large bracketed expressions. If x is a row and kept inline, then the expansion gets really wide. To keep it compact and have the symbols oriented the same as their expansion, you'd have to put the x above A and that's just silly.

Well, there used to be a tradition in universal algebra (and category theory) of putting functions/operators at the right side of the arguments but it seems to have ultimately died out. And "y = xA" is the standard notation in the linear coding theory, even to this day: message vectors are bit strings, not bit columns.

I think there’s also some other pretty big field that tends to put the vector on the left… statistics? Economics? For some reason I couldn’t find any links, so maybe I just got this from, like, one random statistician. If only they’d told me about sample sizes.

Your reasoning starts out nicely with y=Ax, but you get the wrong conclusion. The layout of x and y are not affected at all by row or column order, they are just consecutive numbers in either case. So you have to look at the layout of A.

For the row-major layout the multiplication Ax ends up being more cache-efficient than for the column major layout, as for calculating each component of y you need to scan x and a row of A.

I’m pretty sure BLAS will tile out the matrix multiplication internally anyway, so it doesn’t matter. At least for matmat would, is matvec special?

However matmat is done, row-major vs column-major for both matrices shouldn't make a difference (for square matrices at least).

I don't know if tiling is done for matvec. I don't think it makes sense there, but I didn't think about it too hard.

y = x @ W + b

Is how you'd write it in NumPy and most (/all?) deep learning libraries, with x.shape=[batch, ndim]

I'd personally prefer that to be the convention, for consistency.

I think in math columns as vectors is a more common representation. Especially if we talk about matrix multiplication and linear systems.

Hang on, doesn’t numpy use C (row major) array ordering by default? checks docs Seems like it does. However, numpy array indexing also follows the maths convention where 2D arrays are indexed by row, column (as does C, by the way), so to access pixel x, y you need to say im[y, x]. And the image libraries where I have toyed with with numpy integration (only Pillow, to be honest) seem to work just fine like this — a row of pixels is stored contiguously in memory. So I don’t quite see why the author claims that numpy stores a column of pixels contiguously, but I have only glanced at the article, so quite probably I have missed something.

You're right. Numpy stores arrays in row-major order by default. One can always just have a look at the flags (ndarray.flags returns some information about the order and underlying buffer).