I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day.

DTrace is used extensively in FreeBSD development these days.

It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/

My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.

We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.

I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.

I use dtrace often, but am too busy to write articles about it, sorry. A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.

I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel.

For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.

You really aught to look at Bryan Cantrill's twitter feed then. It happens frequently enough that he notices blog posts and tweets them.

Then again Cantrill has an axe to grind.

In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill

I'm no fan of the man, but it doesn't mean that dtrace is not an excellent tool.

We use it every day at Circonus. One of us is learning something new about how our stuff works every single day.

Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX.

It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.

I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same:

    /pattern/ { statements }

In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations).

So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.

This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.

> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

Then you haven't been looking:

http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi...

http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor...

https://www.joyent.com/blog/node-js-in-production-runtime-lo...

https://twitter.com/b0rk/status/681536144362848257

https://speakerdeck.com/sartak/dtrace-war-stories

https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/

https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...

Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace

Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)