Do fused multiply-add operations for matrices really matter? There are two relevant cases I could come up with.

1. You have large matrices. In this case, I'd think that the O(n^2) addition can be ignored, because it gets dwarfed by the the O(n^3ish) multiplication.

2. You have small matrices. In this case, the computation is likely bound by memory bandwidth. Waiting for the matrices A and B takes most of the time. Then you multiply them, which should go quickly, since the matrices are small. Then you do the addition with the matrix C, but since the product AB is already in cache and since you'd have to wait for the matrix C anyway, there is not much to be gained with a fused multiply-add.

You can have your cake and eat it. E.g. C++ expression templates allow you to fuse application of multiple operators.

>You can have your cake and eat it

Yes, but it's C++, so the cake has 200 ingredients, a 1000 step recipe (the majority of which steps are deprecated and have newer alternatives), tastes bland, and takes days in the oven.

Yes that's the plus of using a language that supports every single paradigm ever invented. I hear that they'll add S-expressions in the next version. [1]

Jokes aside, that is pretty cool, I didn't know C++ templates did that.

1: btw Lisp could do that as well, if they wanted. C++ supports thousands of syntax features, Lisp supports infinite.

How can you? Genuinely asking. You can overload operator + or operator *, but how could you make a dedicated ternary (three-way) operator that combines those two operations into something like std::fma? I guess you could overload the function call operator, but that doesn't offer much over just defining a function.

You define operators on your e.g. Vector types to construct a type that represents lazy evaluation of the expressions and reflect the operations at the type level. So for a * b + c you would get a type like Add<Mul<Vec, Vec>, Mul>.

The evaluation then happens as assignment or Vec construction when you have something like:

    Vec r = a * b + c

You could either inline the functions on Mul and Add in the evaluation loop and rely on the compiler to deduce that an FMA instruction can be used. In the unlucky case, you at least avoid the intermediate a * b vector.

Another option, since the expression is represented at the type level is to write a compile-time evaluator that replaces patterns like Add<Mul<Vec, Vec>, Mul> by FMA<Vec, Vec, Vec> using templates.

This general idea is the foundation of libraries like Eigen.

Thanks for this response as well. I didn't see it earlier. It seems pretty interesting what you can do with C++ templates.

You can have the result of * not be the actual multiplication, but a MultiplicationExpression. Then, the result of MultiplicationExpression + Matrix is a MultiplyAddExpression.

Finally you overload the dereference operator of MultiplyAddExpression to actually run std::fma. With some template magic, this can all even be done at compile time.

You can look at boost::phoenix and boost::qi to see this taken to a huge extreme, overriding all of the C++ operators to write parsers using regex-like syntax (e.g. `parser = *a | +b;` generates a parser which recognizes "" or "aaa" or "bb", equivalent to the regex "a*|b+").

That makes a lot of sense, thanks. I hadn't even thought about that. I guess you could also have MultiplicationExpression have an implicit coercion operator so you can use it in places where a matrix is expected, which would just compute the multiplication without any corresponding addition. That's pretty interesting.

Plenty of people are choosing Go because they neither have nor want the C++ cake. The OP was explicitly looking for a simple language, and C++ is the very opposite of that concept.

So yes, you can have efficient nice abstractions, but only at the cost of an extraordinarily complex base language.

My point was that operators don't exclude the possibility of fusion. C++ expression templates were just an example. Numba does fusion in Python.

They don't, but require significantly more machinery in the language than just operator overloading (especially if you want it to work at compile time).

Sure. But somebody writes it once and you can benefit from it. If you don't have operator overloading there isn't any chance of doing this.

At any rate, I feel like the goalposts have been moved a bit. You originally said:

But then you'll soon want to do A*B+C, and you'll find that fused multiply/add is a much faster operation than doing a multiplication and then addition, and soon you'll either be writing mul_add(A, B, C) anyway

Which I just wanted to point out that you can fuse these operations with operator overloading as well. I am pretty sure that

or you'll overengineer a complex solution to make A*B return some kind of object that can recognize the subsequent + operation etc.

was a later edit, I don't remember reading it when I first reacted to your comment. If so, doing that is not really nice, since it breaks the flow of the reactions.

At any rate, I think we agree on all the trade-offs.

And typing the multiplication operator in HN syntax sucks :).

> I am pretty sure that

> or you'll overengineer a complex solution to make AB return some kind of object that can recognize the subsequent + operation etc.

> was a later edit, I don't remember reading it when I first reacted to your comment. If so, doing that is not really nice, since it breaks the flow of the reactions.

I'm honestly not even sure. I had this in mind from the beginning, but I may have initially had a different vaguer comment about it and edited, I'm not sure. I sometimes edit comments right after posting them and then don't leave a note like I did for the later edit of the *s, since I assume no one read them. If this happened, I apologize.

Either way, yes, I think we are mostly in agreement. I'm not actually a big fan of Go's minimalism, so I actually see the value of such constructs.

Yeah, happens to me sometimes as well. Writing starts thinking, leads to more writing.

RETVRN TO GEOMETRY

https://en.wikipedia.org/wiki/Planar_ternary_ring

If you look at the section to do with geometry, you'll see that Fused Multiply-Add T(a,m,c) is the value of y given x where y=mx+c, where the latter is clearly just a line that's not parallel to the y-axis. You may thank me now.