There are some important things to note about these benchmarks. On the Mandelbrot set, I noticed that the program does not use arrays despite the obvious 2D-array structure of the bitmap. Two of the four cores were also underutilized. Next, the k-nucleotide, reverse-complement, and fasta programs only utilized a single core, compared to the C programs utilizing multiple cores. It is not clear to me why this should be the case, but it leads me to cast doubt on the tests I listed.

Simple -- those are the programs that have been contributed.

https://benchmarksgame.alioth.debian.org/play.html

> You will probably come across people saying that the programs are not idiomatic ("enough").

This seems to describe me. I am tempted to rewrite those programs more idiomatically. Thank you for pointing this out.

Please do contribute programs to show your notion of idiomatic!

Isaac, thank you for your (almost :-) indefatigable work for and defence of the benchmarks game.

You're welcome.

I have cut back: x64 single core & x86 quad core are no longer measured, x86 single core updates are not frequent, and x64 quad core updates are less frequent than they used to be (although more frequent than the chart dates might suggest).

I've kind-of been hoping the hardware would fail -- but someone would probably gift me a new machine :-)

From my perspective "defence of the benchmarks game" is a matter of saying "Keep it in perspective" and "Keep it in context" until we do ;-)

One big difference is the rules on pointers.

In C, if you write code like this:

  void dostufftotwoarrays(int* a, int* b);

Then maybe b is equal to a + 3, so your compiled code has to assume whenever it writes to an element of a, it might have changed an element of b, and so reload b from memory. Alternatively, it can do a bunch of tests at the start of the function, and compile multiple copies of the function for different situations.

In FORTRAN, this isn't possible (you don't tend to pass around raw pointers), so it is much easier for the compiler to reason about what objects might be in the same place, and which aren't.

The restrict keyword goes some way to fixing this in C, but is tricky yo use correctly, and also doesn't solve all problems.

Your Fortran summary is incorrect. F77-style arrays are generally passed as a raw pointer (under the hood.) F90-style allocatable arrays are passed as an array descriptor containing size information, again under the hood.

This doesn't affect aliasing in Fortran at all. Aliasing is simply prohibited, which leaves the compiler free to generate code that gets the wrong answer, if the programmer screws up and has actual aliasing at run-time.

Apologises, you are right. It's so long since I did Fortran I had misremembered that the language made it impossible to alias, as opposed to it being undefined behaviour.

What makes it tricky to use, and where does it fall short?

There are some weird rules, like:

     mypair* restrict position   = &object->pair;
     float*   restrict position_x = &position->x;

Is undefined behaviour, as you can't take a restricted pointer to a child of a restricted pointer in the same scope (the rule isn't quite that simple, but that's part of it). In general, rules about pointers to pointers get complicated, and I've had trouble with them in the past.

In general, it's a good idea to try to only use restrict for "leaf" pointers, not intermediate objects, but that does block some optimisations.

There is also the obvious problem of people getting carried away, and marking something 'restrict' where they shouldn't have.

While 'restrict' is very easy to use on simple structures (like just int*), if you look around you'll find very little, if any, material about using it for complicated nested data structures.

There's also the fact that restrict is only a promise to the compiler "this function will not be called with aliased arguments", not also a command "do not allow people to call this function with aliased arguments".

As far as I know Fortran, at least in its original form, enforced the latter (that was fairly easy to do, as it didn't have dynamic memory allocation, pointers or recursion (the compiler determined the location in memory of all variables))

Neither of your options is the actual rule. The actual rule is that when the program runs, there can't be any runtime aliasing.

You can totally call a subroutine with several arguments being the same thing, as long as you don't do anything bad inside the subroutine.

I'm aware of this because I tried to get the PathScale compiler team to add a warning for incorrect aliasing. That wasn't possible. But I did get a flag which made the compiler obey C aliasing rules for Fortran, and it was part of the "got the wrong answer? run with these flags and if you get the correct answer, your program is not standard-conforming" thingie.

Please educate me and explain how that contradict anything I claimed, as I don't see it.

I claim it is only a promise, you claim you can call it with identical arguments. I don't see a contradiction; making a promise isn't the same as keeping it.

Similarly, your argument that it is fine to break that promise as long as you don't do anything bad inside the subroutine, I read as "you sometimes can get away with breaking a promise", but again, that's not the same as not making the promise.

What am I missing?

The difference is subtle. Here is a (stupid) example:

    int chooseone(restrict int* a, restrict int* b, bool c) {
       if(c) return *a; else return *b;
    }

Here, we can pass the same pointer for a and b, because only one of them is ever accessed in any call to the function.

Thanks. I reread http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf, and initially didn't get that from the informal description of restrict given there. It says (section 6.7.3):

"An object that is accessed through a restrict-qualified pointer has a special association with that pointer. This association, defined in 6.7.3.1 below, requires that all accesses to that object use, directly or indirectly, the value of that particular pointer."

So, it hinges on the meaning of 'accesses'; it is a runtime thing, not a compile-time thing.

  > I would expect two compiled languages [...] to perform almost exactly the same when they're doing nothing but arithmetic operations [...]

Different languages are going to have different memory models which is going to result is vastly different performance. E.g. a GC vs reference counting vs linear types. If you are doing arithmetic on something that needs to be managed (e.g. an array), you'll see differences.

Some language features close the door for optimizations. E.g. the ability to dynamically add or remove methods to an object/class can make method dispatch more or less expensive.

Finally, some languages have undefined behavior (e.g. what happens when there's an overflow or a division by zero) while other languages want specific behavior for the edge cases. This all affects the performance of the resulting code, even for basic arithmetic operations.

These are all good points.

In this case though, C doesn't have garbage collection or OO features, and (I believe) has unspecified behavior for arithmetic exceptions. I assume Fortran does the same (since it's comparably fast), so I guess I still don't really see where people expect the difference to come from.

C and Fortran both obey IEEE 754 and have well-defined arithmetic exceptions on most platforms -- but programmers usually instruct the compilers to ignore all that in the interests of faster code.

As for what the differences are, try talking to a compiler engineer. The better ones love how Fortran features like the argument aliasing rule for 100% of code makes generating fast code easier. In fact that's one of my tests for compiler engineers in interviews; if they've worked on a C/C++/Fortran compiler and can't say anything good about the challenges and opportunities of Fortran optimization relative to C/C++, then they haven't really drilled down on performance issues!

Lies, damn lies, and benchmarks.

I know as a long-time C programmer, there are small changes I can make to source that will result in big-time performance gains.

Could hinge on how tight their FORTRAN code is vs their C code...

Could also depend on how much love their respective machine code translations have been getting.

Fortran hasn't been all caps for nearly 30 years now...

It is very easy for a physical scientist to write Fortran that performs well - the syntax looks very similar to the maths that is being modeled. It is a lot harder for a scientist to write tight C code.

Not bashing Fortran here, I'm defending it.

> small changes … result in big-time performance gains.

Look and you might see measurements of several different C or C++ or … programs doing the same task with different performance.

https://benchmarksgame.alioth.debian.org/u64q/performance.ph...

Like you observe, the compilers matter.

Historically, C compilers just couldn't do such a good job at optimising, because C is pretty complex and has loads of undefined behaviour. Also, Fortran compilers have always been optimised for numerical computation.

Recently * , a lot of effort has gone into making C compilers produce very optimised code. In general, this is a good idea because humans are bad at optimising and also remembering every damned Intel vectorisation instruction.

So historically, Fortran was much faster, recently * with compiler advances in e.g. reordering and AMD64, it's possible C compilers beat Fortran compilers.

* edit: adjust your definition of recently to encompass the entire timeframe of Fortran's existance

Compilers are afraid to optimize around possible UB, even though they can.

Having worked with key people from the SGI compiler team, and then competed with Intel / PGI / Sun / gcc on x86, I totally disagree.

I guess I should've looked when I said "Compilers". I only skimmed two and one of them is LLVM.

I totally see UB as a blessing in disguise for optimizer.

Usually, yeah. The art is making it debuggable, when the user accidentally strays into UB and needs help not doing that.

C can be faster than Fortan if you optimize it so. Fortran is built to run at very close to the max C speed for these things out of the box (as I understand it after doing some extensive googling a while back).

> I would expect two compiled languages with reasonable compilers to perform almost exactly the same when they're doing nothing but arithmetic operations

Auto-vectorization? These implementations vary wildly between GCC and Intel.