I wouldn't be surprised if there's something similar happening here.
I like PyPy as an example: on the surface, implementing a Python runtime in Python and expecting performance gains seems crazy. PyPy manages to outperform CPython because although a C implementation should theoretically be faster, realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach.
I don't know C or Rust well enough to comment on these specific scenarios, but if two technologies can be fast, and one makes that speed accessible while the other encourages implementors to leave performance on the table, that's much more useful information to me than seeing a back-and-forth where folks compete to implement optimizations I'll never make in my own projects.
Pypy outperforms cpython for the simple reason that pypy is a jit where cpython is a basic bytecode interpreter. Anything else is icing on the cake.
The only reason python seems slow as an implementation language is because it was traditionally slow; the reason it was traditionally slow is that cpython, the only major implementation, is slow. Common lisp, for instance, is similarly dynamic (moreso, in fact), also is also generally natively compiled by itself, and is quite performant.
One quick example: standard-class objects in Lisp are mostly more dynamic than python objects[1] but structure-class objects are quite static, and compilers will use this to their advantage.
On top of this, functions in python are in fact objects; in python you can do this just fine:
def foo():
foo.bar = 3
def bar():
print(foo.bar)
There are obviously other ways to accomplish this in lisp, but the fact that someone might do something like this can inhibit optimizations.If you read the Lisp specification you will see dozens of places where they allow for opting out of dynamism in ways that help the compiler optimize. Type declarations are one and the fact that a function defined in a single file is considered to have a fixed definition for the scope of the file is another.
1: It's not quite as easy to add new slots to Lisp standard-class instances on the fly as it is to python class instances, but it's doable, and Lisp, of course, supports class redefinition in a way that few other languages (including Python) do.
https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding...
To put it another way, there's a reason PyPy trails CPython language features on the scale of literal years. This is not explained by the affordances of the language they're written in! They're just projects with very different requirements which, on some platforms, happen to run a lot of similar-looking code.
the reason is under-funding, and the lack of developers to the cause as a consequence, not the portability complexities of jit.
I have nothing against Rust personally, but it's ultimately not an apples-to-apples comparison if they're not implementing the same algorithm, or even if they're not using the same mechanisms (e.g. Rust explicitly using SSE intrinsics, which are certainly available to C in just as idiomatic a fashion).
and you can always compare “equivalent” algorithms as some languages may not be able to efficiently express the same algorithm as another. i know what you are after, but trying to have some benchmark that is “fair” according to some spec that is important to your needs will just be seen as pointless to others. benchmark game at least lets us see what the performance ceiling is of various languages, and anytime you object to a language having a worse implementation you may go fix it.
i think what anyone with a clue understands is that C C++ and Rust all have roughly equivalent performance ceilings. Nobody really thinks Rust is faster than C now.
The submission title would beg to differ.
(I know, it explicitly calls out "benchmarks" as the context.)
I think languages like Rust or Swift have significant advantages around safety over C/C++, while not sacrificing much in terms of performance. But if one language's benchmark contributors are willing to put in more effort than another's to eke out additional performance, then you're going to see skewed results in favor of whichever has the more fervent evangelists or whichever language has more to prove.
If the goal is to compare performance of two languages which can express the same optimization in exactly the same way, and only one uses it, then the benchmarks fail in that respect.
no single person can maintain optimally implemented solutions over dozens of languages. its up to we the people to help out. you want perfectly equivalent comparisons, make it happen
You can have "soft" or you can have "warm" — which do you want?
You can have an apple or you can have an orange — which do you want?
Yep. There are good reasons why this set of benchmarks is called The Computer Language Benchmarks Game.
Not if it includes a JIT compiler, as PyPy does. I don't know much about PyPy, but if most of the time is spent in JITted machine code, the fact that it's written in Python may not affect performance much.
Not really: It's the compilation strategy (that whole meta-tracing JIT compiler thing, compared to a simple bytecode interpreter) that makes the difference, not the surface syntax of the implementation language - which is actually not 'real' Python, but a restricted, less dynamic subset known as RPython.
Also note that the CPython is deliberately kept 'dumb'.
1. http://www.ats-lang.org/ 2. http://web.archive.org/web/20121218042116/http://shootout.al... 3. https://stackoverflow.com/questions/26958969/why-was-the-ats...
What is "real theorem-proving language" in this context? ATS has ATS/LF that is designed for writing formal proofs in the language.
http://ats-lang.sourceforge.net/DOCUMENT/INT2PROGINATS/HTML/...
In this case it seems like benchmark code is allowed to use intrinsics, which can degenerate into a situation where a benchmark in language X is more "glorified x86 Assembly code" than actual code in language X.
This is not very useful for comparing languages IMO. Especially since all of Rust, C, C++ can use this strategy and become almost identical in both code and performance.
if you want to know something more than that, like how performance tends to end up after inputs of identical effort and talent you will need to recruit a lot of people and money to run really large scale experiments and when you are done people on HN will just endlessly find nits to pick with your results whenever they don’t agree with their preconceived notions
It is not idiomatic Haskell at all and loses all of the touted benefits.
Is there a separate benchmark that only accepts idiomatic code?
One can, with sufficient effort, essentially write C code in Haskell using various unsafe primitives. We would argue that this is not true to the spirit and goals of Haskell, and we have attempted in this paper to remain within the space of “reasonably idiomatic” Haskell. However, we have made abundant use of strictness annotations, explicit strictness, and unboxed vectors. We have, more controversially perhaps, used unsafe array subscripting in places. Are our choices reasonable?
http://www.cs.ru.nl/P.Achten/IFL2013/symposium_proceedings_I...
But when using that variant all the correctness guarantees that Haskell is known for are no longer on the table.
Also C was known for not being a fast language on 80's home hardware, ruled by Assembly.
So low level coding for OS isn't coding like C and is just the way for mechanical sympathy, regardless of the language.
Also C++ and Fortran don't have to spend much effort to achieve that.
Am I missing something?, is there a list somewhere of the people who have contributed to this?
These benchmarks are almost always either useless or a scam - you either end up writing rewriting the same implementation n times or you don't utilize the capabilities of the language, either way you're not really measuring much of anything intrinsic to the language itself - Rust and C both have the same backends and if you really care about performance you're going to take it to the max anyway, so inference by the compiler isn't that important.
• https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
• https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
• https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
It's possible that someone has already submitted both algorithms for both languages, and different approaches won for language-specific reasons.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
None of them have SSE intrinsics or are quite as long as the Rust version.
I find it doubtful that SSE intrinsics wouldn't help the C version, if they are indeed helping the Rust version. This seems fairly easy to check as the Rust version has a non-SSE fallback code path - I'd do it myself but am not able to at the moment.
Well, it looks like the submission process[0] and the maintainer[1] do, actually.
[0]: https://www.reddit.com/r/rust/comments/kpqmrh/rust_is_now_ov... [1]: https://www.reddit.com/r/rust/comments/kpqmrh/rust_is_now_ov...
"Registration with @gmail.com addresses is blocked due to abuse, please register with another address. You can change it later, once signed up and verified."
However, apparently, you can register using a Google, Bitbucket or GitLab.com sign-in.
In addition to what you've already mentioned, if I'm not mistaken, the programs also take different multi-threading approaches. One of the tasks in this benchmark involves reversing three very large strings. My C program took a fairly simple approach and does that reversal process for each string serially but processes multiple strings in parallel. The Rust program on the other hand does the opposite and does the reversal process for each string in a parallel manner but only processes strings one at a time. The algorithm the Rust program uses is more complicated which results in a considerably larger program size that also is a bit less flexible regarding the input (it assumes each line of input is 60 characters whereas my program will work with lines of any length) but it results in better CPU utilization, less clock time, and less memory use. I've been meaning to submit a faster C program using this faster algorithm but have simply been too busy with other things.
If I had to write a performant library at work, I too might rely on CPU-specific assembly wrappers in my code. But IMO, such code has no place in a general-purpose cross-language benchmark site.
My guess is that other people would take your "performant library at work" premise as justification for including that code.
C code that pretends it does not need to care about it's platform is not idiomatic, it's just suboptimal.
- Code that most C books/courses would teach you how to write
- Portable C code (arguably portability is one of C's biggest successes!)
- Code that you'd expect to find in the K&R book
All high level languages are portable, including some older than C.
It helps when one drags the OS the language was born on, along via parallel standards like POSIX.
Theoretically portable, maybe. But C is actually portable in practice - you can compile C code for the vast majority of CPUs and OSs.
Every “implementation defined” feature (https://en.wikipedia.org/wiki/Unspecified_behavior#Implement...) creates two different languages (examples are sizes of basic numeric types, signedness of the char type, and evaluation order of arguments)
Also, the standard library is so limited that, historically, you needed lots of stuff that wasn’t standardized between systems (https://en.wikipedia.org/wiki/GNU_Autotools: “It can be difficult to make a software program portable: the C compiler differs from system to system; certain library functions are missing on some systems; header files may have different names.”)
But yes, bringing up C on a system is a lot easier than doing the same for, to pick another extreme, Ada, but that’s because only half the job is done. Autotools does a bit of the rest, but still requires help from program writers.
Right, with C, you may have to fix a few bugs to get a program working properly in a new system due to the reasons you mention.
But with many other languages, you'd have to port a compiler and the standard library first, which is probably harder :)
Original C was created after UNIX already existed, used for UNIX's first rewrite and until UNIX source code left Bell Labs, it only worked on PDP-11 computers.
http://cm.bell-labs.co/who/dmr/chist.html
> Thompson was faced with a hardware environment cramped and spartan even for the time: the DEC PDP-7 on which he started in 1968 was a machine with 8K 18-bit words of memory and no software useful to him. While wanting to use a higher-level language, he wrote the original Unix system in PDP-7 assembler. At the start, he did not even program on the PDP-7 itself, but instead used a set of macros for the GEMAP assembler on a GE-635 machine. A postprocessor generated a paper tape readable by the PDP-7.
> Although we entertained occasional thoughts about implementing one of the major languages of the time like Fortran, PL/I, or Algol 68, such a project seemed hopelessly large for our resources: much simpler and smaller tools were called for. All these languages influenced our work, but it was more fun to do things on our own.
> By 1970, the Unix project had shown enough promise that we were able to acquire the new DEC PDP-11. The processor was among the first of its line delivered by DEC, and three months passed before its disk arrived. Making B programs run on it using the threaded technique required only writing the code fragments for the operators, and a simple assembler which I coded in B; soon, dc became the first interesting program to be tested, before any operating system, on our PDP-11. Almost as rapidly, still waiting for the disk, Thompson recoded the Unix kernel and some basic commands in PDP-11 assembly language. Of the 24K bytes of memory on the machine, the earliest PDP-11 Unix system used 12K bytes for the operating system, a tiny space for user programs, and the remainder as a RAM disk. This version was only for testing, not for real work; the machine marked time by enumerating closed knight's tours on chess boards of various sizes. Once its disk appeared, we quickly migrated to it after transliterating assembly-language commands to the PDP-11 dialect, and porting those already in B.
> By 1971, our miniature computer center was beginning to have users. We all wanted to create interesting software more easily. Using assembler was dreary enough that B, despite its performance problems, had been supplemented by a small library of useful service routines and was being used for more and more new programs. Among the more notable results of this period was Steve Johnson's first version of the yacc parser-generator [Johnson 79a].
> By early 1973, the essentials of modern C were complete. The language and compiler were strong enough to permit us to rewrite the Unix kernel for the PDP-11 in C during the summer of that year. (Thompson had made a brief attempt to produce a system coded in an early version of C—before structures—in 1972, but gave up the effort.)
What failed was the initial port attempt to C, not UNIX running on PDP-7.
While I am quite critic of C, I take the effort to learn and criticise properly, because regardless of my opinion it isn't going away on my lifetime and bills need to be paid.
Yes, but that is the key bit right, the fact that C didn't have structures yet is what caused the effort to be aborted, and once those were added they succeeded.
> Original C was created after UNIX already existed, used for UNIX's first rewrite and until UNIX source code left Bell Labs, it only worked on PDP-11 computers.
Regardless of failed porting effort, the original UNIX, implemented in Assembly and running on the PDP-7, was already being used by a small community.
Practically all non-trivial C codebases have platform specific segments. They can be either isolated to their own libraries or wrapped around tons of preprocessor switches.
C code is portable in the sense that it's not too much work to get a C compiler to work on a new hardware platform. On the other hand, compiling a large real-world C codebase on the said platform will probably require porting, and the effort may or may not be trivial.
These are the kind of things that make working in one language different from another.
I'd actually like to see the SSE version in C so I can compare the two implementations and how much grief you have to go through.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
On a quick inspection:
- The Rust code is about twice as long.
- The Rust code has CPU feature detection and SSE intrinsics, while the C code is more idiomatic.
- The lookup table is larger in the Rust code.