Five years ago I tried to make some sense of this by researching all of the existing technologies. In the kernel I found:

   - ftrace (https://lwn.net/Articles/290277/)
   - tracepoints (https://www.kernel.org/doc/Documentation/trace/tracepoints.txt)
   - kprobes (https://www.kernel.org/doc/Documentation/kprobes.txt)
   - events (https://www.kernel.org/doc/Documentation/trace/events.txt)

   - BPF, a packet filter that grew into a tracing framework
     (https://lwn.net/Articles/599755/)

   - perf
   - systemtap
   - lttng
   - other, random, fragmented things

I don't mean to put down anybody's work, but this stuff will never be user-friendly as long as it remains so fragmented, IMHO.

https://jvns.ca/blog/2017/07/05/linux-tracing-systems/

- Some of those are not in-tree, like LTTng and SystemTap.

- Tracepoints, kprobes, events, and uprobes are all event libraries used by perf or ftrace, just like DTrace had multiple providers (fbt, pid, etc).

The real fragmentation is perf and ftrace, since both are in-tree front ends. That's not too bad, and they both have different strengths.

eBPF is weird in that it's neither an event library or a front end. It's programmatic capabilities. We're mostly using an out of tree project, bcc, to run it.

If there's just a variety of tools for the same task, then that's healthy competition and how you get better software.

If no single tool can fulfill all your tracing needs, that's still not necessarily a condemnation of the tools. It's entirely possible that each tool can complete a subset of tasks, but is significantly simpler to use as a result, so SUM(effort to learn tools you need) may still be comparable to the effort of a theoretical omni-tool.

[1]: https://github.com/iovisor/bcc

But ultimately the reason that this is the current state is because of how Linux is developed. Trying to push something like Jails or Zones is an exercise in futility because the patchset would be too large, would touch everything, and the infrastructure would likely not be reusable by other people.

Previous view: Linux tracing is so complicated. Without a personally important usecase I probably won't invest the time to learn it.

Current view: Linux tracing is so complicated. Without a personally important usecase I probably won't invest the time to learn it.

Soo... What I can say is I really hate these keyboard sounds. Otherwise I'm not sure I learned much.

Seems like there could be a lot of opportunity, hopefully I'll get a chance to dive in and find out myself.

This book would _probably_ get you moving in the right direction: http://www.brendangregg.com/sysperfbook.html

It should be something like: look at your bottlenecks and utilization, look at your costs, look at (cost effective) ways to reduce or remove those bottlenecks or that utilization. Pick the cheapest place to have a bottleneck. Using SSD at an extra 30$ a month lets you use half the CPU and RAM, saving 60$ a month? Go for it.

I... haven't tried it in ages. I've no idea if it works. It used to crash my system easily, but maybe now it's fine.

Maybe this could be solved by weighting samples by the inverse of number of running threads at the time

[1]: https://lwn.net/Articles/630965/