Mathematician here. I never heard that.

(In many branches the idea is that you care about the abstract linear transformation and properties instead of the dirty coefficients that depend on the specific base. I don't expect a mathematician to have an strong opinion on the order. All are equivalent via isomorphism.)

Thank u so much brother.

Not a mathematician, but programmers definitely don't agree on whether matrices should be row-major or column-major.

I'm surprised we even agree that they should be top-down.

At this point might as well make them match the x/y convention, with first index increasing to the right, and second index increasing from bottom to top.

Programmers don’t agree on the x/y convention.

Most fields of math that use matrices don't number each element of the matrix separately, and if they do there will usually be two subscripts (one for the row number and one for the column number).

Generally, matrices would be thought in terms of the vectors that make up each row or column.

Not a mathematician, just an engineer that used matrix a lot (and even worked for MathWorks at one point), I would say that most mathematicians don't care. Matrix is 2D, they don't have a good way to be laid out in 1D (which is what is done here, by giving them linear indices). They should not be represented in 1D.

The only type of mathematicians that actually care are: - the one that use software where using one or the other and the "incorrect" algorithm may impact the performance significantly. Or worse, the one that would use software that don't use the same arbitrary choice (column major vs row major). And when I say that they care, it's probably a pain for them to think about it. - the one that write these kind of software (they may describe themselves as software engineer, but some may still call themselves mathematicians, applied mathematicians, or other things like that).

Now maybe what the author wanted to say is that some language "favored by mathematician" (Fortran, MATLAB, Julia, R) are column major, while language "favored by computer scientist" (C, C++) are row major

ah, so you can get row vectors with a type cast in C but not column ones. While in Fortran and friends is the converse (if they casted). Yep that is more mathy. Linear map evaluation is linear combination of columns.

EDIT: type cast or just single square bracket application in C

Languages that don't have multidimensional arrays tend to have "arrays of arrays" instead, and that naturally leads to a layout where the last subscript varies fastest. Languages that do have multidimensional arrays can of course lay them out in any order.

I'm a mathematician. It's kind of a strange statement since, if we are talking about a matrix, it has two indices not one. Even if we do flatten the matrix to a vector, rows then columns are an almost universal ordering of those two indices and the natural lexicographic ordering would stride down the rows.

Yes. I think what all mathematicians can agree on is that the layout (and the starting index! :-) is like this:

  A[1,1] … A[1,n]
   …        …
  A[m,1] … A[m,n]

I assume they're talking about how when multiplying by a matrix by a vector, it's more natural to think of the matrix as a collection of columns than as a collection of rows, and to think of the vector as a column vector.

That layout is a nearly universal convention in applied practice such as statistics. Readers would be very very confused if you flipped it the other way.

The irony is that "programmers" are much more divided on this than statisticians are.

It depends which side you are doing the multiplication on? Most linear algebra textbooks work matrix-vector, where the vector is a column vector. In that arrangement, the resulting vector is formed by dot products of the rows of the matrix with the vector.

On the other hand, you see vector-matrix multiplication a lot in other places, for example, the Markov chain literature. There, the vector is a row vector and the resulting vector is formed by dot products of the columns of the matrix with the original vector.

What I suspect he really means is that FORTRAN lays out its arrays column-major, whilst C choose row-major. Historically most math software was written in the former, including the de facto standard BLAS and LAPACK APIs used for most linear algebra. Mix-and-matching memory layouts is a recipe for confusion and bugs, so "mathematicians" (which I'll read as people writing a lot of non-ML matrix-related code) tend to prefer to stick with column major.

Of course things have moved on since then and a lot of software these days is written in languages that inherited their array ordering from C, leading to much fun and confusion.

The other gotcha with a lot of these APIs is of course 0 vs 1-based array numbering.

The MKL blas/lapack implementation also provides the “cblas” interface (I’m sure most blas implementations do, I’m just familiar with MKL—BLIS seems quite willing to provide additional interfaces to I bet they provide it as well) which explicitly accepts arguments for row or column ordering.

Internally the matrix is tiled out anyway (for gemm at least) so column vs row ordering is probably a little less important nowadays (which isn’t to say it never matters).

Oh yes, from an actual implementation POV you can just apply some transpose and ordering transforms to convert from row major to column major or vice-versa. cblas is pretty universal though I don't think any LAPACK C API ever gained as wide support for non column-major usage (and actually has some routines where you can't just pull transpose tricks for the transformation).

Certain layouts have performance advantages for certain operations on certain microarchitectures due to data access patterns (especially for level 2 BLAS), but that's largely irrelevant to historical discussion of the API's evolution.

> is written in languages that inherited their array ordering from C

It’s not just C. Modern GPU hardware only supports row major memory layout for 2D and 3D textures (ignoring specialized layouts like swizzling and block compression but none of them are column major either). Modern image and video codecs only support row major layout for bitmaps.

I'm an applied mathematician and this is the most common layout for dense matrices due to BLAS and LAPACK. Note, many of these routines have a flag to denote when working with a transpose, which can be used to cheat a different memory layout in a pinch. There are also parameters for increments in memory, which can help when computing across a row as opposed to down a column, which can also be co-opted. Unless there's a reason not to, I personally default to column major ordering for all matrices and tensors and use explicit indexing functions, which tends to avoid headaches since my codes are consistent with most others.

Abstractly, there's no such thing as memory layout, so it doesn't matter for things like proofs, normally.

Recently graduated math student here. The definition of the "vec" operator which turns a matrix into a vector works like this, stacking up columns rather than rows.

https://en.wikipedia.org/wiki/Vectorization_(mathematics)

Really wonder who in his right mind, who can also read a book, would decide to do so in order to enumerate the sequences of items…

Vectors are traditionally written as columns, so just writing all the columns in left-to-right order into a bigger column makes sense. The confusing bit isn't the ordering of the elements in a matrix, it's that someone decided to write vectors as columns to begin with!

The plus side is that you can use row vectors with matrices acting from the right, and keep the traditional matrix multiplication. Rare case of two wrongs creating a right.

It took me long while to stop being afraid when people wrote vec^T where ^T means transposed. I was like "but why can we do that", and the mathematician answered - "well, because it is just more convenient this time". o_O

This is one of those things that's perennially annoying in computer graphics. Depending on the API you can have different conventions for:

- data layout (row-major vs. column major)

- pre-multiplication vs. post-multiplication of matrices

Switching either of these conventions results in a transposition of the data in memory, but knowing which one you're switching is important to getting the implementation of the math right.

And then on top of that for even more fun you have:

- Left- vs. right-handed coordinate systems

- Y-up or Z-up

- Winding order for inside/outside

There are lots of small parity things like this where you can get it right by accident but then have weird issues down the line if you aren't scrupulous. (I once had a very tedious time tracking down some inconsistencies in inside/outside determination in a production rendering system.)