News for hackers. Stuff that matters.
Re:Column Vectors vs. Row Vectors by captainmuon on Thursday October 27, 2022 @03:06AM (#33354206)
I studied physics, and I did a bit of 3D graphics back in the day, and I could never remember the difference between column and row vectors (or co- and contravariant vectors, bra and ket vectors, ...) and when to use which. My guilty secret is that I don't care.

For me a vector is just abstractly three or four numbers together. (Maybe with some defined transformation properties if I'm doing physics.) Whether a matrix is stored row-major or column-major is an implementation detail I try to hide.

In fact I used to make indexing errors all the time. But now when I need to deal with vectors and matrices, I look up or decide on the "correct" way, hide everything behind Vector and Matrix classes, implement dot and cross product once, and never fall back to the raw vector elements if possible.

Re:Column Vectors vs. Row Vectors by chombier on Thursday October 27, 2022 @03:22AM (#33354316)
If one thinks of vectors as the matrix of some linear map over the scalars (and similarly covectors as linear maps to the scalars), and assuming the usual composition order of maps then it seems there is no choice as to the column/row representation: vectors have to be (n x 1) and covectors have to be (1 x n)

This doesn't say anything regarding their practical storage, but at least conceptually vectors/covectors are a bit more than just a collection of numbers.

This distinction is pretty useful in practice when encoded in types to avoid implicitly mixing things that are conceptually different.

Re:Column Vectors vs. Row Vectors by IIAOPSW on Thursday October 27, 2022 @04:01AM (#33354586)
I was with you until you said bra and ket didn't do it for you. Its right there in the name bra(c)ket. <bra|, |ket>. Logically equiv to any other notation, but impossible to fudge up. When they are multiplied in a way that lines up like <bra|ket>, you got an inner product (scalar measure of overlap). When they are multiplied wrong way around, you got a ketbra |i><j| (which is basically a neat way to represent a matrix element at indices i and j). Is the bra the row and the ket the column or vice versa? Don't matter. Just keep things facing whichever way they were originally facing and nothing will go wrong. In fact, you don't even need to think in terms of columns at all if you don't want. That's it. That's the whole system.

For the engineers who never seen this, |i> |j> |k> work exactly like unit vectors i,j,k, but you aren't limited to the usual 3 dimensions and 3 letters. Writing |x> in a ket like this just means "I am using x as a label for a degree of freedom in some vector space, not as the literal value x". This lets you do cool shit, like if you got n dimensions you can write a vector as sum v_i |i> ranging from 0 to n. You can also do super useful stuff like merging two vector spaces together to represent a larger system (aka a tensor product). Eg if I have a vector v = sum v_i |i> representing one subsystem and another vector w = sum w_j |j> representing some other subsystem, I can construct the vector u = sum v_i w_j |i, j> representing the whole system (sum is over both i and j).

(I know you already learned all this, I wrote it for everyone else's benefit).

Unrelated, given the webs origin at cern, HTML should have used bras as start tags and kets as end tags. <HTML| <head|head> <body| ... |body> |HTML>.

Re:Column Vectors vs. Row Vectors by azalemeth on Thursday October 27, 2022 @07:28AM (#33356014)
Fun anecdote: when I was a teenager learning LaTeX, I'd got in the habit of google image searching the symbols – "Latex infinity" would turn up a pic of an infinity symbol with "\infty" below it; "latex contour integral symbol" would show "\oint", and so on.

As a nineteen year old first-year university student, I genuinely typed "latex bra" into google image search, pressed enter, and then had a second or two of shock as my brain was genuinely surprised to not see quantum mechanics in front of me with a nifty little "\langle" involved, but rather lots of women in decidedly unique looking underwear...

Re:Column Vectors vs. Row Vectors by sam-s on Thursday October 27, 2022 @12:39PM (#33360207)
I learned that "garbage collection" is not just a memory management technique but also refers to big trucks removing the content of trash cans when I took a "nerd test".
Re:Column Vectors vs. Row Vectors by dbcurtis on Thursday October 27, 2022 @02:12PM (#33361616)
One of the architects of the Pentium Pro, Intel’s first out-of-order execution engine, told of walking up to a Coke machine with an “out of order” sign on it — “Oh, out-of-order has other meanings.”
Re:Column Vectors vs. Row Vectors by IIAOPSW on Friday October 28, 2022 @06:58PM (#33379040)
The vending machine at intel drops a can of coke. A CPU architect walking by picks it up then inserts his money. The machine is out of order.
Re:Column Vectors vs. Row Vectors by sfpotter on Thursday October 27, 2022 @12:01PM (#33359563)
If you define u to be a column vector, and u' to be its (conjugate) transpose, and define matrix multiplication as usual, then u'v is a dot product and uv' is an outer product. What you mentioned about the standard basis vectors in R^3 works for standard basis vectors e_i in R^n using this notation; e.g., e_i e_j' is an all zero matrix with a one in the ith row and jth column.

If you have a hard time remembering whether u should be a column vector, you can simply identify things that look like u with vectors (|ket>), and things that look like u' with linear functionals (<bra|). Then u'u is just a dot product.

There are other notations, too... Sometimes (u, v) and u \cdot v are inner products, sometimes u \otimes v is an outer product.

Generally linear algebra-style notation is terser and has less line noise. IMO, some manipulations are simpler to think about using this notation than with the bra-ket notation, especially when you're doing numerical optimization or numerical linear algebra.

Maybe another reason to prefer it is that modern notation for operator algebra follows this notation and not the bra-ket notation (although I think bra-ket notation was invented for this purpose...). If L is a linear operator, L^* is its adjoint, Lu is the application of L to u, etc...

Of course, what notation you should use is determined by what community you belong to. If you're publishing papers on numerical linear algebra, it would be very strange to decide to use the bra-ket notation instead. The purpose of notation is communication. The goal is to communicate ideas to other researchers. It's good to develop the ability to be flexible with notation and go along with what other people are doing.

Re:Column Vectors vs. Row Vectors by captainmuon on Friday October 28, 2022 @12:51PM (#33373990)
I am actually a fan of bra-ket notation. But in my mind, <i|j> is the amplitude for going from state i -> j. My point is, if you just have one state that you want to talk about, which one do you write down <a| or |a>? It doesn't matter. You only write the brakets when you want to calculate transition amplitudes/inner products, or cross products.

I agree about just keeping them the way they are originally facing and nothing will go wrong. The nice thing is that you can mechanically rearrange things almost like regular algebra, i.e. draw out a common factor, and don't have to think what these objects are (a vector? a dual vector? a functional?).

Same with four-vector notation / Einstein summation. As long as you are following the rules for raising and lowering indices, you cannot form "wrong" products or mess up the transformation properties.

Said in another way: as a (former) experimentalist, I don't care if the vector is p^mu or p_mu. All I care about is the product p^mu p_mu.

Re:Column Vectors vs. Row Vectors by ogogmad on Thursday October 27, 2022 @05:24AM (#33355199)
I imagine that's the right way to do it. Treating a matrix as a table of numbers is too concrete, and there are far too many conventions for how to do it. Abstractly, it's a type-(1,1) tensor, which is naturally isomorphic to a linear map. (I'm slightly abusing how I'm using the term "naturally isomorphic"). There are many notations for tensors that don't mention indices, including the somewhat surprising Abstract Index Notation by Roger Penrose.

For the sake of completion: A vector is a type-(1,0) tensor. A linear form (or "covector" if you prefer this term) is a type-(0,1) tensor. A bilinear form is a type-(0,2) tensor.

The point of saying the type of a tensor, is to clarify what geometric object an array of numbers represents. A curse (and a blessing) of matrix notation is that it doesn't tell you this. Both a type-(0,2) and type-(1,1) tensor can be represented by matrices, but the first is used for (let's say) dot products, and the second is used for transformations.

The only problem with type-(m,n) notation is remembering whether "m" is the contravariant order or the covariant order. It turns out that it's contravariant, which is the opposite to what the naming would suggest. In programming, we would do {cov: m, con: n}, so we wouldn't have to remember.

Re:Column Vectors vs. Row Vectors by geokon on Thursday October 27, 2022 @07:28AM (#33356013)
Unfortunately this physics-guy/spherical-cow thinking is really common in people doing linear algebra on computers and the results are poor performance. Abstractions are great for general programming, but when you are doing tight-loop math it's really not applicable. I've seen this a lot recently with "clever" NDArray style math libraries (not to mention the utility of more than 2 dimensions is .. very limited)

In my not-super-extensive experience, if you're programming and dealing with linear algebra problems - and you're past the MATLAB prototyping stage - then I really suggest using the BLAS/LAPACK API. You're not gunna beat decades of nerds programming ballistic missiles on punchcards. They're kinda weird and unwieldy and don't match to textbook math - but they've had a ton of thought put into them. And are made with as few footguns as possible. At least re-implementing an algo with them a few times is really educational. You'll see that it forces you to think about your memory layout and you'll realize that the back of napkin complexity calculations are actually tricky to massage into a good algorithms that use memory well. The final result with something like the Intel MKL will be way way faster than anything your abstraction can achieve

(Unfortunately I haven't found a good primer on using the BLAS well)

Re:Column Vectors vs. Row Vectors by halotrope on Thursday October 27, 2022 @07:33AM (#33356050)
"You're not gunna beat decades of nerds programming ballistic missiles on punchcards" funny and true
Re:Column Vectors vs. Row Vectors by sfpotter on Thursday October 27, 2022 @08:20AM (#33356451)
MATLAB uses BLAS and LAPACK under the hood. You can write reasonably fast code in MATLAB if most of your time is spent calling out to faster low level implementations.

There are also many different implementations of BLAS, with varying performance. The API itself is not the reason for its performance. Actually, I would argue that it’s a pretty crufty API which should probably be done away with at this point and replaced with something less awkward. I don’t know how many times I’ve had to remind myself what a “leading dimension” was…

A good example is BLIS, which does provide the BLAS API but has a more modern “object-style” C API which is significantly easier to work with than the BLAS API with no performance hit.

Also, depending on what you’re doing, abstractions can be very helpful indeed to keep around. Even at the BLAS level, groups of bits are thought of as singles, doubles, real or complex… a matrix has a size and shape… etc. Having this information is useful. Linear algebra is full of type information which can be used to dispatch different algorithms.

Lots of other alternatives to BLAS which are a bit higher level but still very useful: Eigen and Armadillo in MATLAB, Julia, etc.

Anyway, I would say (based on my actual, significant experience ;-) ) people are beating the guys with punch cards all the time, no reason to stay in the 70s…

Re:Column Vectors vs. Row Vectors by CrazyStat on Thursday October 27, 2022 @09:20AM (#33357080)
> MATLAB uses BLAS and LAPACK under the hood. You can write reasonably fast code in MATLAB if most of your time is spent calling out to faster low level implementations.

Many years ago I was implementing a statistical model in R that involved a lot of matrix multiplication (the model included a non-negative matrix factorization step). Fed up with how slow it was in R, I quickly reimplemented it in MATLAB--literally copy-pasted the code and changed things like operators and function names--for an immediate 5x speedup.

Re:Column Vectors vs. Row Vectors by geokon on Thursday October 27, 2022 @09:33AM (#33357216)
"The API itself is not the reason for its performance"

It eliminates many performance mistakes by making it so that when you do something dumb it's generally immediately apparent. This mostly boils down to nonsensical/slow operations being usually impossible and any memory copying being explicit.

I can only tell you my own experience. Going through the steps of transforming a few "textbook algorithms" into BLAS made me think about the problems much more clearly and in a way I would have missed using a higher abstraction.

All the examples you give (Matlab, eigen, armadillo, Julia) allow you to really easily write really bad code (blas under the hood is kinda irrelevant). But you're totally right that they're useful. If you want to call out to code that does an SVD or something simple that exists in a library then they're generally just fine and you can't mess it up too much.

I use BLAS with a repl from Clojure (thanks to neanderthal) so it's very painless. Sounds like you're using it in a compile/run loop from C/C++ which sounds like very much not fun.

Re:Column Vectors vs. Row Vectors by sfpotter on Thursday October 27, 2022 @11:18AM (#33358854)
Here's BLIS's object API:

https://github.com/flame/blis/blob/master/docs/BLISObjectAPI...

Searching "object" in BLIS's README (https://github.com/flame/blis) to see what they think of it:

"Objects are relatively lightweight structs and passed by address, which helps tame function calling overhead."

"This is API abstracts away properties of vectors and matrices within obj_t structs that can be queried with accessor functions. Many developers and experts prefer this API over the typed API."

In my opinion, this API is a strict improvement over BLAS. I do not think there is any reason to prefer the old BLAS-style API over an improvement like this.

Regarding your own experience, it's great that using BLAS proved to be a valuable learning experience for you. But your argument that the BLAS API is somehow uniquely helpful in terms of learning how to program numerical algorithms efficiently, or that it will help you avoid performance problems, is not true. It is possible to replace the BLAS API with a more modern and intuitive API with the same benefits. To be clear, the benefits here are direct memory management and control of striding and matrix layout, which create opportunities for optimization. There is nothing unique about BLAS in this regard---it's possible to learn these lessons using any of the other listed options if you're paying attention and being systematic.

Re:Column Vectors vs. Row Vectors by rpep on Thursday October 27, 2022 @09:00AM (#33356819)
On top of that, you almost certainly do not want to instantiate an N x N matrix, because sparsity is an important factor in most real problems.
Re:Column Vectors vs. Row Vectors by pfortuny on Thursday October 27, 2022 @09:28AM (#33357175)
They should not be calles vectors both.

Horizontal vectors are forms (maps from R^n to R). Vertical vectors are vectors (elements of R^n).

Forms applied to vectors are “rows multiplied by columns”, and so: numbers.

Same thing as bras and kets, by the way.

Re:Column Vectors vs. Row Vectors by lupire on Thursday October 27, 2022 @11:33AM (#33359138)
This is a great example of how math is generally weak at types (constantly doing implicit casts and conversions with ambiguous notation, all the way back to Liebniz notation for calculus, and probably further), and the computer scientists add clarity.

It's also an example of how "making it simpler" by dropping types (old Python, Bash, typeless vectors and matrices) ends up making it harder to get work done right when the ideas are non-trivial.

Re:Column Vectors vs. Row Vectors by pfortuny on Thursday October 27, 2022 @01:23PM (#33360906)
Well, yes. But it is another way of saying that we think in analogies.

At the same time, forms are also vectors… and vectors are forms of the dual… So it is not that easy.

Re:Column Vectors vs. Row Vectors by captainmuon on Friday October 28, 2022 @01:07PM (#33374213)
If you have two pointy arrows, and want to know if they are perpendicular, then you use the scalar product u⋅v and check if it is 0. There is no difference between u and v in this case. You could say v is a vector, and u a function taking a vector and returning a scalar (a form, a dual vector, ...). But often that is unneccessary artifice. You can just say they are both vectors in the same vector space, which is imbued with an inner product.

Declaring that some objects are forms (duals, covariant vectors, ...) is only neccessary when you want to keep track of transformation properties, for example.