News for hackers. Stuff that matters.
Re:A Large Scale Study of Programming Languages and Code Qua... by luu on Tuesday November 04, 2014 @04:54PM (#8559297)
The claim:

“Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages.”

But how did they determine that?

The authors looked at the 50 most starred repos on github, for each of the 20 most popular languages plus typescript (minus CSS, shell, and vim). For each of these projects, they looked at the languages used (e.g., projects that aren’t primarily javascript often have some javascript).

They then looked at commit/PR logs to determine figure out how many bugs there were for each language used. As far as I can tell, open issues with no associated fix don’t count towards the bug count. Only commits that are detected by their keyword search technique were counted.

After determining the number of bugs, the authors ran a regression, controlling for project age, number of developers, number of commits, and lines of code.

That gives them a table (covered in RQ1) that correlates language to defect rate. There are a number of logical leaps here that I’m somewhat skeptical of. I might believe them if the result is plausible, but a number of the results in their table are odd.

The table “shows” that Perl and Ruby are as reliable as each other and significantly more reliable than Erlang and Java (which are also equally reliable), which are significantly more reliable than Python, PHP, and C (which are similarly reliable), and that typescript is the safest language surveyed.

They then aggregate all of that data to get to their conclusion.

I find the data pretty interesting. There are lots of curious questions here, like why are there more defects in Erlang and Java than Perl and Ruby? The interpretation they seem to come to from their abstract and conclusion is that this intermediate data says something about the languages themselves and their properties. It strikes me as more likely that this data says something about community norms (or that it's just noise), but they don’t really dig into that.

For example, if you applied this methodology to the hardware companies I’m familiar with , you’d find that Verilog is basically the worst language ever (perhaps true, but not for this reason). I remember hitting bug (and fix) #10k on a project. Was that because we have sloppy coders or a terrible language that caused a ton of bugs? No, we were just obsessive about finding bugs and documenting every fix. We had more verification people than designers (and unlike at a lot of software companies, test and verification folks are first class citizens), and the machines in our server farm spent the majority of their time generating and running tests (1000 machines at a 100 person company). You’ll find a lot of bugs if you run test software that’s more sophisticated than Quickcheck on 1000 machines for years on end.

If I had to guess, I would bet that Erlang is “more defect prone” than Perl and Ruby not because the language is defect prone, but because the culture is prone to finding defects. That’s something that would be super interesting to try to tease out of the data, but I don't think that can be done just from github data.

Re:A Large Scale Study of Programming Languages and Code Qua... by ggreer on Tuesday November 04, 2014 @08:26PM (#8560065)
You've done a good job of pointing out weaknesses in the paper, but I have a question about your defect rate argument: Would you hold the same position if the study had said the opposite? In other words, if it claimed that Ruby and Perl had more bug fixes (and thus defects) than Erlang and Java, would you claim the study was flawed due to a culture of meticulous bug-finding in Perl and Ruby?

From Conservation of Expected Evidence[1]:

If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree.

It really seems like a stretch to say that higher bug fix counts aren't due to higher defect rates, or that higher defect rates are a sign of a better language. Language communities have a ton of overlap, so it seems unlikely that language-specific cultures can diverge enough to drastically affect their propensity to find bugs.

1. http://lesswrong.com/lw/ii/conservation_of_expected_evidence...

Re:A Large Scale Study of Programming Languages and Code Qua... by loup-vaillant on Wednesday November 05, 2014 @04:53AM (#8561123)
Personally, I have read the abstract alone, and was pleased to see it nurtured my own preferences: static typing is better, strong typing is better. But then I saw this disclaimer:

> It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size.

So I dismissed the paper as "not conclusive", at least for the time being. I wasn't surprised if their finding were mostly noise, or a confounding factor they missed.

By the way, I recall some other paper saying that code size is the most significant factor ever to measure everything else. Which means that more concise and expressive languages, which yield smaller programs, will also reduce the time to completion, as well as the bug rate. But if their study corrects for project size, while ignoring the problems being solved, then it overlooks one of the most important effect of programming languages in a project: its size.

Re:A Large Scale Study of Programming Languages and Code Qua... by lmeyerov on Tuesday November 04, 2014 @08:41PM (#8560104)
I'd love to see a methodology similar to the one here analyzing concurrency bugs: http://www.cs.columbia.edu/~junfeng/09fa-e6998/papers/concur... . Akin to what's done in social sciences, they applied simple labels to bugs in the bug repos -- a grad student and some undergrads can label a lot in a couple weeks -- and regress on that.
Re:A Large Scale Study of Programming Languages and Code Qua... by Animats on Wednesday November 05, 2014 @12:38AM (#8560650)
What's striking is the comment that the "mysql" project has the highest bug fix density of the C programs. That seems unexpected, because MySQL is very heavily used and reasonably stable.

It may be simply that MySQL bugs actually get fixed. The work behind this paper counts bug fixes, not bug reports. Unfixed bugs are not counted.

Re:A Large Scale Study of Programming Languages and Code Qua... by gmac on Wednesday November 05, 2014 @07:11AM (#8561433)
Having used MySQL in the past and vowed never to use it again, I would not be especially surprised if the MySQL code base was unusually buggy. On the other hand, high use rates could well lead to high bug discovery rates.
Re:A Large Scale Study of Programming Languages and Code Qua... by mixonic on Wednesday November 05, 2014 @08:41AM (#8561687)
Just the initial data-set alone is going to be incredibly mis-representative of software in general.

* Most-starred projects mean these are all successful projects. Much software is not successful.

* A successful project means there are likely more-experienced-than-average engineers coding.

* Most-starred projects will be older, more stable code-bases than most software.

* Open Source development is a small slice of software development.

* Github is a sub-set of Open Source development.

And I expect there are a myriad of holes to poke elsewhere. In general I distrust any research that surveys GitHub and tries to make claims about software development in general. It is lazy.

Re:A Large Scale Study of Programming Languages and Code Qua... by mtdewcmu on Tuesday November 04, 2014 @10:24PM (#8560390)
>> why are there more defects in Erlang and Java than Perl and Ruby?

I have no experience with Erlang, but one reason I'd expect Java to have more defects than Ruby and Perl is that Java is more verbose, i.e. it takes more code to get something done. One would naively expect to find an association between the size of commits and their propensity to contain errors.

Re:A Large Scale Study of Programming Languages and Code Qua... by mheiler on Wednesday November 05, 2014 @03:30AM (#8560952)
The nice thing about the paper is that they try to put data where others put guesses, rants, expectations, and believes. Arguing with data is actually difficult (and rare) in the software engineering field.

The practical result is that language choice doesn't matter since the effects are very small.

Re:A Large Scale Study of Programming Languages and Code Qua... by zvrba on Wednesday November 05, 2014 @03:04AM (#8560913)
> There are lots of curious questions here, like why are there more defects in Erlang and Java than Perl and Ruby?

Maybe because Erlang and Java are used for projects of higher complexity (larger scope, more interacting components, etc.)? Did the authors try to address this issue at all?

Re:A Large Scale Study of Programming Languages and Code Qua... by pdpi on Wednesday November 05, 2014 @06:48AM (#8561394)
The standout result for me that led me to believe this was very likely the case was that Erlang scored _horribly_ on concurrency bugs. To me, that makes sense: you're seeing all the bugs that come from trying to tackle tricky high concurrency situations, which is why people picked Erlang in the first place. If people tried to tackle those exact same problems in other languages, we'd probably see them doing worse at concurrency.
Re:A Large Scale Study of Programming Languages and Code Qua... by lostcolony on Wednesday November 05, 2014 @12:02PM (#8562781)
Not to mention the Erlang mantra of 'let it fail'. They're tracking bugs, not severity. Someone may file an issue "hey, in this instance, this thing goes wrong", but because of the supervision process, it doesn't actually cause anything to break (just a logged error message and a bit of system churn). The language actively encourages you to code happy path, and address failure conditions only as necessary, relying on the supervision tree to handle errors that stem from deviating off the happy path.