News for hackers. Stuff that matters.
Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @12:31PM (#15373628)
>, kernel developers still use email because it is faster than any of the alternatives. Over the course of the last year, the project accepted about eight changes per hour — every hour — from over 4,000 developers sponsored by over 400 companies

Compare with Google's ~1875-per-hour from 2015[1]:

>, and Google’s 25,000 engineers make about 45,000 commits (changes) to the repository each day.

Obviously, we can't extrapolate that Google should switch to an internal SMTP project management system layered on top of Perforce because "email is faster than any of the alternatives". Same for other companies with high commit rates to their source repos like Microsoft and Facebook.

Since Linus & Linux community never did an A-vs-B experiment of email vs open-source-JIRA-clone as a project workflow, I think we can only safely say that email works best for a disparate group of volunteers with different employers. Email is the lowest common denominator that everyone could agree to use.

You couldn't get 4000 opinionated developers to agree to use an open-source JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud.

Therefore, listing advantages of email clients such as "threaded tree views" etc is actually hiding the underlying reason email-as-workflow works for Linux. Email isn't about "throughput"; it's "acceptance".

[1] https://www.wired.com/2015/09/google-2-billion-lines-codeand...

[2] https://en.wikipedia.org/wiki/Application_lifecycle_manageme...

Re:Why kernel development still uses email (2016) by codemac on Saturday September 30, 2017 @01:28PM (#15373941)
I got in this same discussion with gregkh on Reddit, and Torvalds in person. Linux development is much slower than many large software engineering firms, because the gatekeeper model has many more integrators and reviewers as a proportion of the members than the "pull request"/ team code review model

Acceptance is certainly part of it, but 8 changes per hour is 8-10 email threads per hour, which is just not that hard to track as an individual, thus no need to change anything. Right now at work on a large (200ish) team, I get cc'd on about the same amount of patches.

Re:Why kernel development still uses email (2016) by abiox on Saturday September 30, 2017 @02:50PM (#15374414)
> Compare with Google

isn't google's situation a monorepo composed of many distinct projects? there doesn't seem much utility in the comparison.

> Email isn't about "throughput"; it's "acceptance".

this seems to be the similar, in the other direction. using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.

Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @03:10PM (#15374529)
>using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.

I don't think that equating it from the reverse direction works. Here's why:

Using the article's count of ~4000 developers, let's consider all software companies that employ 4000+ developers as part of a "natural experiment"[1]. This includes companies and organizations like MS, Facebook, NASA, SAP, Oracle, etc.

Each company "votes" in the marketplace of ideas to coordinate software development for a complex project. One company might use JIRA. Another uses MS Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP emails similar to Linux kernel development.

Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible.

Since no single software company I know of uses SMTP emails as their ALM workflow, it either means that (1) companies are wasting money & losing iteration velocity by avoiding email -- or -- (2) the Linux kernel community has special circumstances that makes using a true unified ALM unrealistic. I believe that special circumstance is: 4000 programmers work for different employers instead of a single employer.

[1] https://en.wikipedia.org/wiki/Natural_experiment

Re:Why kernel development still uses email (2016) by abiox on Saturday September 30, 2017 @03:36PM (#15374650)
> Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible.

in the abstract these are generally viewed as or presumed to be positive, but the pressures and objectives of a business can be rather complex and other things will often compete for finite resources.

having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering.

outside the sales pitches, it's also not clear how an alm interacts with notions of 'cost-efficienty' and 'velocity'. introducing process and bureaucracy isn't always a win.

perhaps the presumed predilection of large organizations for alms can be found in alms addressing (or seeming to address) other needs, pressures or requirements.

Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @04:02PM (#15374744)
>having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering.

Sure, I agree this is true in a general sense (Dilbert cartoons, Peter Principle, etc). However, for things like choosing technology to help with market superiority, companies (especially software engineering companies) will eventually gravitate to something better.

We could argue that companies are "inefficient" for not embracing remote work-from-home instead of open offices, etc. (general inefficiencies). But, even badly run companies don't insist on using inferior floppy drives instead of USB drives or wireless network file transfers. Yes, a law firm might limp along with an 15-year old version of Microsoft Word 2000, but a software engineering company will not. (technology inefficiencies)

> introducing process and bureaucracy isn't always a win.

The Linux dev community also has process & bureaucracy. See my previous link for the strict and precise formatting rules to submit kernel patches in emails. That's process. Just because it's not in a web form with fields doesn't mean it's not "process". They're just using emails as the workflow for it. (Linus has famously chastised contributors for not following "the rules".) Process & bureaucracy is not bad thing -- its effort just has to match the scope of work being delivered. (E.g. using JIRA for a "hello world" toy program is overkill.) The Linux kernel community seems to have converged on the level of bureaucracy that works for them.

It's interesting that even young YCombinator companies with much less than 4000 employees also don't stay with SMTP email-as-ALM as they get bigger. If email-as-ALM was a better software dev philosophy, there would be a huge arbitrage opportunity to outcompete everybody. Instead, the email-as-ALM fits Linux kernel devs' specific circumstances.

Re:Why kernel development still uses email (2016) by makapuf on Saturday September 30, 2017 @04:34PM (#15374862)
having worked in large and small orgs, what I can say is, sure, a large org has many more inefficient workflows and latency than a smaller one. However it also has much more drive than the N independent smaller orgs it needs to accomplish the same scope. As a whole, a bunch of 10 person startups is generally more inefficient to accomplish something a big enterprise can do. (and of course, to accomplish what a startup can do, a big org won't be able to do for ages. Formula one vs huge train.) My point being, dont compare a big organization with a small team. Compare a small team within this org with an independant team. The indeêndent team wins, but not by far. With several competing small startup, the thing is, you cannot see the inefficiencies.
Re:Why kernel development still uses email (2016) by marcosdumay on Saturday September 30, 2017 @03:41PM (#15374672)
Or, alternatively, ALMs provide some other kind of value (that is iteration velocity or efficiency) that Linux developers don't care about.
Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @04:11PM (#15374785)
>Or, alternatively, ALMs provide [...] iteration velocity [...] that Linux developers don't care about.

I agree and that's why the author quoting Greg Kroah-Hartman about commit velocity (throughput of "8 changes per hour") in his article is actually a distraction from answering the actual question as stated in the article's title. The real reason for kernel devs use of emails isn't the throughput.

Re:Why kernel development still uses email (2016) by walkingolof on Saturday September 30, 2017 @01:22PM (#15373892)
Its not comparing the same thing I think, commits to a repo and diffs sent over email, and even if it was, its not really the same thing anyway...

Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.

Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @01:28PM (#15373939)
>Kernel development seems to progress fine,

I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.)

I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.)

Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.

Re:Why kernel development still uses email (2016) by wmf on Saturday September 30, 2017 @02:46PM (#15374387)
It seems like that argument cuts both ways; what about developers who refuse to use the email-based process?
Re:Why kernel development still uses email (2016) by jasode on Saturday September 30, 2017 @02:53PM (#15374433)
>; what about developers who refuse to use the email-based process?

I guess I don't understand your question. In Linux, the email-as-the-workflow[1] has been agreed upon from Linux Torvald and his lieutenants all the way down.

I suppose that if a one refused to use email-based-workflow, they'd have to find a Linux kernel maintainer to be a champion for their proposed changes and submit them on their behalf. (Is this a common scenario? I don't know.)

If a rebel Linux kernel programmer insisted on staging changes for review in Gerritt/JIRA/github/etc, he's going to be isolated in an island because he's not part of the agreed-upon workflow: emails.

[1] https://github.com/torvalds/linux/blob/master/Documentation/...

Re:Why kernel development still uses email (2016) by wmf on Saturday September 30, 2017 @03:03PM (#15374490)
If Linus declared that everyone must use, say, Gitlab from now on, some Linux developers might leave. But developers who don't use email may never start working on Linux. That's why it cuts both ways: either option alienates some developers.
Re:Why kernel development still uses email (2016) by walshemj on Saturday September 30, 2017 @08:00PM (#15375648)
A developer that does not use mail :-) that is very unlikely
Re:Why kernel development still uses email (2016) by dlp211 on Saturday September 30, 2017 @09:07PM (#15375878)
A developer that does not use mail for code submission.
Re:Why kernel development still uses email (2016) by bassman9000 on Saturday September 30, 2017 @03:59PM (#15374731)
4000 developers sponsored by 400 companies.

Let than sink in. It's not only 4000 opinionated devs. It's, potentially, 400 completely different organizations; 400 management layers on top; 400 different priority sets; devs in every time zone; etc. Google is just 1 company (a dozen, if you want to break down Alphabet).

It's amazing that they manage to get 8 changes an hour, and still be one of the most successful open source, and software project in general.

Why changing the system?

Re:Why kernel development still uses email (2016) by erikbye on Saturday September 30, 2017 @04:29PM (#15374843)
14000 developers from 1300 companies have contributed, since 2005.

From 3.19 to 4.7, an average of 4600 lines of code was added daily.

Over 80% of all kernel development is done by developers paid for their work.

A small number of companies are responsible for a large portion of the total changes. These being, from biggest to lowest contributor: Intel (12.9%), Red Hat (8,0%), Linaro (4%), Samsung, SUSE, IBM, Google (2%), AMD, Texas Instruments, ARM, Oracle.

Contributions from unpaid developers have been in decline for many years. 14.6% in 2012, 7.7% in 2016.

Re:Why kernel development still uses email (2016) by lifeisstillgood on Sunday October 01, 2017 @02:49AM (#15376743)
Can I ask how you got that info? Are committers listed as paid / unpaid ?
Re:Why kernel development still uses email (2016) by erikbye on Sunday October 01, 2017 @03:45AM (#15376834)
Re:Why kernel development still uses email (2016) by emmelaich on Saturday September 30, 2017 @09:14PM (#15375889)
The great thing about email is that it is asynchronous.

The problem with a ALM is it is a global lock. It is enormously disruptive if it is unavailable or even slow.