Let's consider now why C is a great language. It is commonly believed that C is a hack which was successful because Unix was written in it. I disagree. Over a long period of time computer architectures evolved, not because of some clever people figuring how to evolve architectures---as a matter of fact, clever people were pushing tagged architectures during that period of time---but because of the demands of different programmers to solve real problems. Computers that were able to deal just with numbers evolved into computers with byte-addressable memory, flat address spaces, and pointers. This was a natural evolution reflecting the growing set of problems that people were solving. C, reflecting the genius of Dennis Ritchie, provided a minimal model of the computer that had evolved over 30 years. C was not a quick hack. As computers evolved to handle all kinds of problems, C, being the minimal model of such a computer, became a very powerful language to solve all kinds of problems in different domains very effectively. This is the secret of C's portability: it is the best representation of an abstract computer that we have. Of course, the abstraction is done over the set of real computers, not some imaginary computational devices. Moreover, people could understand the machine model behind C. It is much easier for an average engineer to understand the machine model behind C than the machine model behind Ada or even Scheme. C succeeded because it was doing the right thing, not because of AT&T promoting it or Unix being written with it.
I've spent some time working on VLIW architectures where C is a terrible abstract representation of the CPU's inner workings. Get anything done efficiently was awful and required huge, cumbersome, carefully structured intrinsics. Or raw assembly.
Is this is fault of VLIW? Not really. Its mostly the reality of a world that would rather run C code than use VLIW architectures. So we don't have VLIW or other CPU architectures that don't work well with C.
Back in th 8 and 16 bit days of the home computer systems, I never felt the need to use C.
I could touch all the hardware using Assembly, Turbo Pascal.
Apple was using Object Pascal for system programming and a few friends were doing Amiga games with Amos.
Sure I did learn C at high school, but it only became a requirement for the university work that had to run on UNIX systems.
"people could understand the machine model behind C" -> that is why every substantial C program is ridden with undefined behavior, overflows, memory leaks, race conditions....
"C succeeded because it was doing the right thing" -> If the right thing is giving rise to the software exploit industry and causing billions in damages..
I suggest you read Richard Gabriel's "Worse is Better" (http://www.jwz.org/doc/worse-is-better.html) and forget anything that Stepanov has to say on the matter. Either he is utterly clueless or a dangerous imbecile.
C has been in use for decades now. We only have to look at the facts, not listen to fallacies that various cretins feel the urge to proclaim.
The x86 string functionality that I'm aware of was directly derived from C string functionality, and any decent library of course uses them. You don't directly map to those underlying opcodes because that would be silly, and would completely undermine any platform independence you might have.
The same for vectorization. You don't explicitly express vectorization in your code, but of course all decent C compilers can easily and robustly generate such code.
C isn't close to the hardware (beyond very high level notions like "contiguous memory"). But it's a simple enough language that it's very heavily optimizable.
From what I've seen auto-vectorization is anything but robust, often you need non-portable annotations to get anything decent. See e.g.: http://locklessinc.com/articles/vectorize/ where there's an interesting case where using "if" instead of a ternary operator is enough to prevent optimization.
The string instructions tend to be slower than using generic instructions, especially in the presence of SSE.
I once wrote a SIMD C++ template library based on valarray: http://www.pixelglow.com/macstl. Sadly it has languished over the years but I would love to work on it again. It had novel (at the time?) vectorized trigonometric functions, for example.
The simplicity of C means you can't make many assumptions of how it is handled. Same problem in C++. You can make a lot of optimizations to assembler if you can guarantee a value won't change, or if the relative location in memory needs to remain static because anything can rip the address and do awful pointer arithmetic with it.
I like to compare to asm.js - it is a reduced language in that it restricts the environment from doing things it can't easily optimize away, except in that case its to assembler. Reduced set versions of C exist to do the same already, but it would be nicer if you just had a fast language default with straightforward rules with ways to just declare wonderland functionality that can mess things up that means the compiler needs to preserve the machine assumptions rather than the language structures.
Why is that so? Do more modern languages need another twenty years before they can compete with C (and eventually beat it performance-wise) ?
When I started coding in the mid-80's C compilers were dog slow and no better than the alternatives (Pascal, PL/I, Modula-2, Cedar, ...).
In fact, game developers would talk about high level languages the same way they talk about current modern languages vs C/C++, and use nothing else other than Assembly.
However with UNIX spreading into the enterprise and bringing C along, it meant compiler vendors focused on optimizing for the language they were getting money for.
The main issue when people compare languages, is that they forget although the design drives the implementation, not all implementations are alike.
ps: for instance in some cases, jitted code will go faster than C, some Jitted kernels (never tested personally) allowed for a good 30% performance increase.
"Not that fast" doesn't necessarily mean that it isn't the fastest we have got.
These days the most important factor to performance, in almost all programs, is memory accesses.
Writing cache-friendly code is made relatively easy by C. You control the placement of your data into aligned cache lines. You control the order in which the cache lines are accessed.
The cost of the computational instructions is usually swallowed by the memory misses.
C's common ABI (for x86/64) now passes up to 6 parameters via registers, and doesn't unnecessarily spill to the stack. Additionally, link-time optimizations now allow cross-compilation-unit inlining which can optimize away the ABI costs.
Of course, a C program doesn't have to use C strings, but so many libraries and APIs do (including the C standard library) that they are unavoidable for most practical purposes.
D eliminates this problem by using dynamic arrays to represent strings. A dynamic array is a pointer/length pair.
Calling conventions can be a bottleneck but this is not unique to C, any other compiled language allowing separate compilation units to be linked together has the same issue. Techniques like LTCG can avoid it.
But given that C is almost always at or near the top of benchmarks both for size and speed, maybe it's not that much of a bottleneck after all.
C is not that fast. One of the major problems is that it's close to hardware. 1970s hardware that is. Ken Thompson reportedly once said: "I'm not going to do nibbles. I have an 8 bit processor".
A good example of how bad it has become is that modern processors have a rather good understanding of the 'string' concept, and offer instructions to process them. C offers a char*.
Another problem is that C has strict contracts on how parameters are to be passed through, and combined with separate compilation units, this hurts compilers when they try to optimize things.
I see greater potential for a safe higher level language to be able to align closer to modern day hardware than C. Some nice examples: Linear types can avoid garbage altogether, and coroutines can be expressed clearly and correctly using monads. On the other hand, raw performance is rarely needed, and most cycles are burned interpreting things like python and php.