> This is what's wrong with Linux kernel development. We have pluggable subsystems like I/O scheduling or CPU scheduling where BFQ or BFS are not mainlined and have to live as external patchsets. Then we have more than a few competing tracing/probing frameworks all somehow magically in torvalds/linux.git despite their overlap and unfinished (partly experimental) state.

> I used to think that Linux kernel development is about who comes up with good, working code, but more and more I'm starting to get the impression it's that you need to have the right connections for a competing implementation to land.

I think this is true of all sufficiently large organizations with human interactions: politics is an integral part of them.

It also reminds me of Con Kolivas's reasons for no longer working on the mainline kernel: http://forum.digit.in/chit-chat/81361-why-i-quit-kernel-deve....

Generally, I just hope that server developments and improvements trickle down to my personal computing devices.

Lets view this from another angle: you want to work in a project with tens of thousands of developers and literally billions of users. No matter your works technical merits, you simply cannot DEMAND that Linus accepts your huge patchset without having a very good track record.

Con did not have that track record. In fact, he made it very very clear that he was not interested in a long-term commitment and that he was not willing to compromise. I am sure Con was a good programmer, but he was also a cowboy programmer with awful people skills.

So this was never about "knowing the right people".

Let's ignore BFS because my motivator for using a -ck kernel is the availability of BFQ.

I mean, I didn't claim it's generally about knowing the right people, but some occasions, like the one mentioned above, do not leave much room for interpretation. BFQ in particular has been discussed years ago on LKML, and while the general design of it was deemed beneficial, the consensus from the kernel maintainers had been to refactor CFQ and turn it into BFQ. Given the pluggability of I/O schedulers, and already having at least three bundled schedulers, this makes no sense and is hard to explain to users of the kernel. Now years have passed and neither did CFQ evolve in the discussed way nor did BFQ enter torvalds tree.

The argument I'm making is that in the meantime many unfinished and experimental bits landed in mainline. If I were the maintainer of a patch like BFQ, I might possibly take the effort to a more inclusive kernel, and integrate and refine it there after it's mainlined.

Yes, there's two frameworks in the kernel that now have some overlap: ftrace (incl. hist triggers) and perf_events (which can incl. BPF). But it's not a case where they magically got included. Initially they didn't really have overlap: ftrace was doing tracing, and perf_events was doing PMC stats. But they grew over time (especially perf) to the point where there was some overlap. This was noticed and has been discussed, and argued about. Eg, see the later half of https://lwn.net/Articles/442113/. At some point Steven Rostedt, ftrace author, had said:

> Now that perf has entered the tracing field, I would be happy to bring the two together. But we disagree on how to do that. I will not drop ftrace totally just to work on perf. There's too many users of ftrace that want enhancements, and I will still support that. The reason being is that I honestly do not believe that perf can do what these users want anytime in the near future (if at all). I will not abandon a successful project just because you feel that it is a fork.

Hist triggers is an enhancement to ftrace, not its own thing. BPF is a kernel facility originally designed for network I/O, that can also be used by ftrace and perf.

But yes, why not just one unified project? Such a project could bring in the experts from the other external tracing projects too (SystemTap, LTTng, ktap, sysdig, ...). Well, for better or for worse, Linux isn't a company, and Linus isn't its CEO. Scott McNealy, CEO of Sun while DTrace was developed, was fond of the saying:

> all the wood behind one arrow

For an April fool's prank, Sun staff put a giant wooden arrow through his office window. http://www.slideshare.net/brendangregg/from-dtrace-to-linux/...

If Linux was actually Linux Microsystems, the CEO could pick one tracing project and all teams would either toe the company line or leave. But we don't have that.

What we have, including external tracers, is a mess. There's too many tracers. And this becomes a tax on contributors who want to help fix the problem. Which one should you pick? Better lean the pros and cons of them all, which is no easy task.

Fortunately we're getting enhanced BPF, which does everything, so the tracing mess will finally have an end. Hist triggers was a surprise. It would have been a slam dunk before enhanced BPF existed, but now?? I suppose I understand it as an ftrace enhancement, and not its own thing.

That's not the way that Sun worked at all, actually -- and certainly not with respect to DTrace. To the contrary, Sun often deliberately allowed competing projects, believing that the internal competition would generate a better result. (Or, perhaps more likely, Sun simply did not have the command-and-control to prevent such internal competition.) So contrary to your assertion, there was never an executive fiat around DTrace (indeed, there was never much executive fiat at all at Sun), and DTrace had loads of internal competition -- but DTrace succeeded where the others didn't because it was actually useful for solving the problems that others weren't.

There's roughly ten different tracers for Linux. Yes, Sun liked competition, but it did not endure this situation like this -- what project had ten competitors, all staffed and paid for by Sun? If Linux were a company, I cannot believe the situation would have gotten this out of hand.

There were (at least!) ten different kernel debuggers at peak in the late 1990s -- and many different tracers too. (As the saying goes, you haven't heard of them for a reason.) My point is that Sun didn't get out of the morass of myriad half-assed solutions by executive fiat; it got out of it by a small, focused, self-motivated team that was hell-bent on building something that could work on real problems. The nascent success of DTrace had nothing to do with executive management, corporate heft, or marketing; these things only were brought to bear when DTrace had already succeeded where it mattered most: solving real problems.

A nice story, but it's not working out that way for Linux. The Linux kernel engineering talent is there, but split among the different tracers, and, have different priorities to work on. And this has dragged on for years. A rational company could see that it wasn't working (no nascent success), and then combine the right forces to get it done. Linux isn't a company.

You summed it up nicely, thanks. I kinda had LTTng and the other solutions in mind when I said there's many competing implementations, with varying degrees of functionality and maturity, and of course not all in mainline.

As a user, it's a mess because for example Erlang/OTP has added LTTng support, which I expect to be born out of a user requirement based in an environment like SLES or RHEL. That doesn't make it universally practical on a random kernel. Then we have DTrace emulation in SystemTap, and I've heard it kinda works, but I don't know how many probes exist for it.

If I want to trace on illumos or FreeBSD (NetBSD as well), there's just DTrace and I can share tracing code to some extent and the DTrace probes, say, in VMs have to be implemented once.

Like you, I hope there will be a coherent solution, but I don't see it yet.