After perusing the perf man page, the only way I could figure out how to make it accurately count userspace function calls was using hardware breakpoints, e.g.: "perf stat -e mem:0xADDRESS:x"

Obviously that's not a very good approach, because you're limited to the number of breakpoints that your CPU can handle simultaneously (4 on my machine) and there's a lot of overhead. If you know of a better way to accomplish the same thing with perf, I'd be happy to hear it.

https://lwn.net/Articles/499190/ & https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-spa...

http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-upr...

Ah, cool. I just tested it out and it seems to work as documented. Unfortunately it requires root access, and incurs about 1 microsecond of overhead per function call on my machine.

Reading the X-Ray abstract, unless I'm missing it, it doesn't go into detail about the time overhead. I do see mention of CPU and RAM usage, though.

I'm not sure that we have any evidence that X-Ray offers less overhead here, though it's certainly cool if it does.

Can you give an example where perf works with 10 million+ probe points?

:)

(and where this does not have an antagonist effect on the other things on the machine?)

Check out http://www.brendangregg.com/perf.html#DynamicTracingEg

And for those poor people who haven't seen bcc, do yourself a favor and look into it: http://iovisor.github.io/bcc/

Besides, with a unicorn for the logo, whats not to like?