News for hackers. Stuff that matters.
Re:My experience with the computer language shootout by DarkShikari on Sunday April 03, 2011 @06:11PM (#2404278)
In my experience language shootouts in general are often terrible by design -- with these being just a prime example.

A lot of the problem is in the questions they're designed to answer versus the questions people use them to answer.

For example, if I'm comparing Python and C, I typically want to know "how much slower would my program be in Python?", not "how much slower is my program in Python if I spent so much time hyper-optimizing it that I might as well have written it in C?"

But the test cases usually try to answer the latter, not the former.

Re:My experience with the computer language shootout by dkarl on Sunday April 03, 2011 @09:15PM (#2404803)
It might be more reasonable than it seems at first glance. It's true that it's good to know how fast typical code runs, but there's another important question: when I run into performance problems and need to optimize a bottleneck, how fast can I make it before I have to resort to non-portable code or C extensions that complicate my build process?
Re:My experience with the computer language shootout by dagw on Monday April 04, 2011 @04:36AM (#2405840)
Anybody writing, for example, Python code to solve these sort of problems in the real world would instantly reach for numpy. Which, while not part of the core language distribution, is pretty close to being a standard library for most python programmers. I'm sure several of the other languages have similar libraries that are being ignored in these benchmarks. Without taking things like that into account, theses results don't say too many useful things about real world performance.
Re:My experience with the computer language shootout by igouy on Monday April 04, 2011 @03:14PM (#2408062)
Re:My experience with the computer language shootout by dagw on Monday April 04, 2011 @04:05PM (#2408280)
Cool, I missed that
Re:My experience with the computer language shootout by justincormack on Monday April 04, 2011 @06:09AM (#2405976)
Languages/implementations also vary in how much overhead switching to C costs you, especially in loops say, which eg the pidigits benchmark does sort of measure.
Re:My experience with the computer language shootout by davidw on Monday April 04, 2011 @01:22AM (#2405497)
> In my experience language shootouts in general are often terrible by design

My experience is that this is the sort of thing where everyone rags on it, but no one actually attempts to provide something "better".

Re:My experience with the computer language shootout by gaius on Monday April 04, 2011 @06:32AM (#2406008)
Because there's always a "for what" in there. An an analogy, consider MMA. On the surface, it seems to answer the question "what's the best martial art?". But really, it only answers the question of what is the best martial art for fighting a single opponent in an octogon shaped ring in front of an audience aiming for a submission? And the answer is of course Brazilian Ju Jitsu, exactly the answer the founders of UFC wanted...

Former cop Rory Miller writes about this in his book, the police experimented with BJJ and found it useless. Why? Because in BJJ you pin your opponent on his back because it makes a better show for the audience, but as a cop you always pin your opponent on his front so you can handcuff him!

Re:My experience with the computer language shootout by davidw on Monday April 04, 2011 @06:34AM (#2406010)
Ok, sure, but I'd rather see a rough attempt at getting some numbers than just throwing your hands up in the air and saying "gee, that's a hard problem".

Also, I think that we all know enough about programming and languages and their many uses that we can talk directly about it, rather than about an analogy.

Re:My experience with the computer language shootout by akkartik on Monday April 04, 2011 @01:32PM (#2407532)
Why does somebody saying "I wouldn't use this." have to provide an alternative? If such a statement is backed by reasons I find it interesting to read. They're saving me trouble trying it out, just like any other review.
Re:My experience with the computer language shootout by davidw on Monday April 04, 2011 @03:38PM (#2408157)
Why? Because people want to know how fast languages are, and it's not a stupid question if you are considering it in a wider context.

And they're going to do benchmarks.

So you can either complain that they're not good, or you can try and improve them.

Re:My experience with the computer language shootout by akkartik on Monday April 04, 2011 @05:09PM (#2408533)
"So you can either complain that they're not good, or you can try and improve them."

Yes, that sentence is literally correct. But it sounds like it's saying one option is not useful. And I still haven't heard a single reason why reviews of benchmarks are bad.

Responding to a criticism with "those who can't do criticize" is super, super boring. It's been done to death. You're just tarring all criticism with an overly broad brush. If it's bad criticism why is it worth responding to? And if it's plausible criticism why aren't you focusing on the actual details?

Re:My experience with the computer language shootout by davidw on Monday April 04, 2011 @05:53PM (#2408734)
Bitching about language benchmarks has been done to death too.
Re:My experience with the computer language shootout by akkartik on Monday April 04, 2011 @06:06PM (#2408786)
Who are you saying is bitching? OP or DarkShikari or me?
Re:My experience with the computer language shootout by kingkilr on Sunday April 03, 2011 @09:44PM (#2404919)
Maybe there's a space for "if you spent an average amount of time optimizing" :) For example my PyPy optimizations took maybe 4 hours total, with 0 time spent looking at assembly.
Re:My experience with the computer language shootout by igouy on Monday April 04, 2011 @01:44PM (#2407625)
> the questions they're designed to answer

You'd think there'd be some kind of statement about that?

http://shootout.alioth.debian.org/help.php#why

> the questions people use them to answer

You'd think there'd be some kind of advice about that?

http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...

> I typically want to know "how much slower would my program be in Python?"

And we should all know the answer - It depends on how you wrote your program in C and it depends on how you write your program in Python.

Re:My experience with the computer language shootout by akkartik on Monday April 04, 2011 @01:55PM (#2407691)
The question marks seem to suggest that you're poking a hole in grandparent's argument, but it seems like you're both in agreement that the shootout is misused. What am I missing?
Re:My experience with the computer language shootout by igouy on Monday April 04, 2011 @03:28PM (#2408121)
"terrible by design"

Something can be well designed and yet still be misused.

Re:My experience with the computer language shootout by akkartik on Monday April 04, 2011 @05:02PM (#2408509)
'Terrible by design' isn't the same as 'terrible design'. I think he was saying it's deliberately bad for some use case.