Insomuch as I've seen HPC code written by science academics, both when I was one and later when I became an HPC guy, as much as I cringe at some of the Fortran, the C/C++ can be far worse... and I wish that I could retroactively restrict some of my colleagues and customers to sticking with Fortran.

Wow, look at the downvotes! I've been saying this to people for 20+ years, in conference talks and face-to-face conversations, and I've never had a negative reaction. Happy to learn if folks think I'm wrong about my personal experiences. If you've had a different experience, happy to learn about that too (although I'd be surprised if you'd downvote me for that.)

(edit: thanks for the upvotes! and discussion!)

Personally I'd rather see academic c than academic fortran; academic c++ is in the 'oh please no' category.

Yes, I've seen too many unfortunate template libraries, which never work in combination with a different template library.

I've used some quite reasonable academic C++ for geometry processing and meshing.

Note that I'm not saying that science academic C++ is never great; I'm saying that it's often unfortunate. Is the code you're talking about code that's progressed through several maintainers without becoming sad? That's often where things go wrong, not the initial code, but keeping it going, cleanly, through multiple collaborators / post-docs / grad students.

I agree that meshing is definitely a difficult situation; I've never seen a solution (for unusual geometry) that I liked much. Things that can pretend to be rectangular? No problem. Very adaptive meshes? Argh.

Hard to say, but I'm looking at 39 contributors, 539 stars, 3,005 commits [1]. Nicely documented, pretty extensive list of users, CI on Linux and Windows. But with that said, yes, I've also dealt with absolutely horrendous academic C++.

[1] https://github.com/libigl/libigl/

Are there any users that are science academics? I wasn't talking about computer scientists / computer graphics / visualization researchers at all in this thread; they do great with C++. I clicked on a bunch of links of the users of this library and didn't see any physics, astronomy, chemistry, fluid dynamics, etc.

Not sure! We are using it in (non-academic) CFD work.

So it sounds like you're somewhat like the folks I'm talking about, even if the other users of this code are not... but have you had a lot of turnover of your coders, like grad students and postdocs that turn over every 2-4 years? Because that was the full nature of my complaint: science not-computer-scientists over several maintainers doing better in Fortran.

You can also quite simply use local variables, and unroll loops manually:

   for (i ...) {
     double a0 = a[i];
     double a1 = a[i+1];
     double b0 = b[i];
     double b1 = b[1+1];
     c[i] = a0 + b0;
     c[1+1] = a1 + b1;
   }

The hand-unrolling is predicated on the knowledge that the arrays do not overlap. It is obvious to the compiler that the locals do not alias with the arrays, or with each other.

The stores to c[] could alias with a[] or b[], but that hardly matters; we assume that all the array loads and stores take place. We have done ourselves the unrolling that was hindered by the suspicion that the arrays overlap, and could do more of it.

When coding in C for tightness, it helps to pretend that locals (at least basic type scalar ones) are registers and references to arrays and structs are memory loads and stores. Then think like an assembly language programmer, somewhat. It's better to use a few more locals than a few more loads and stores. Try to load once, process with locals, store once.

yes, but restrict is limited in that it can only be used to tell two pointed memories could never alias. Finer grain aliasing can be done using unions and structs, but it's not as flexible as Fortran and the possibility of non-aliasing is easily broken just by allowing address to be taken.

As an aside, how to you accurately measure memory bandwidth? I've been looking for a good solution but couldn't find one.

Some processors expose performance counters but you don't need to accurately measure memory bandwidth to know you are maxing out. Suppose you know that the memory bandwidth is 1Gb/s and you program reads 100Gb of data to produce the result. If the execution takes ~100s then you know you hit the maximum memory bandwidth.

Were you using C99 during your HPC programming in academia?

I used C99 for the computing kernel of one standalone project (simulation of gravitational lensing). Apart from this, it was mostly C++.

What made your work so dependent upon memory bandwidth?

As an introduction, you might want to look into John McCalpin's history as a shallow-water oceanographer and his subsequent fixation on the STREAM benchmark. TL;DR: some algorithms aren't friendly to caching.

I have no idea if this applies to smarnach, but it definitely applies to other people.

John McCalpin did a nice presentation at SC16 giving an overview of memory BW/latency issues in HPC, slides: http://sites.utexas.edu/jdm4372/2016/11/22/sc16-invited-talk...