For a recent example of perf in practice, see "git gc: Speed it up by 18% via faster hash comparisons" on the git mailing list

http://thread.gmane.org/gmane.comp.version-control.git/17228...

Also take a look at sysprof:

http://sysprof.com

The 1.1.6 release uses the same kernel API as perf, but displays full call trees. Screenshot:

http://sysprof.com/screen-shot-4.png

Cool! The sysprof website says that there's a command line tool included, but I can't find it. Do you have any info on that?

Yes, it's called sysprof-cli. It should be built and installed out of the box.

Hmmm, I guess it's not available on Debian wheezy. I just installed sysprof and sysprof-cli wasn't included in it.

I'll install it separately; thanks!

A few other tools I use are atop (more detailed top info, with logging), iostat (harddrive io) and vmstat (memory and cpu details).

As a sysadmin. I don't love any other tool nearly as much as I love atop. It comes close to making all other interactive general-purpose performance monitoring tools redundant. The logging, too, is fantastic. There's nothing like looking through the graphs of my metrics gathered by collectd, wondering why a certain spike or pattern occurred, and being able to step through previous days per-process resource usage via atop and figure it out. There's a great introduction to it at https://lwn.net/Articles/387202/

Do you know of any manual or book-style treatment about using those tools for investigating the performance of Linux systems? Every once in a while I have to look at performance problems on Linux systems, but I always start from zero. I need a place where I can study the topic and refresh myself when necessary, because I'm apparently not capable of mastering it via occasional piecemeal experience.

The closest I've found (and spent 10s, if not hundred+ hours reading) are: System Performance Tuning, 2nd Edition (O'Reilly System Administration) by Gian-Paolo D. Musumeci [1] and the somewhat dated (and Solaris centric) Sun Performance and Tuning: Java and the Internet (2nd Edition) [2]

I'd love to hear that there were better book out there - if anyone has read either of those two, and has found a better one, I'll be one-clicking it in 30 seconds.

[1] http://www.amazon.com/System-Performance-Tuning-OReilly-Admi...

[2] http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/d...

Also don't forget Cachegrind. Cachegrind is amazing and runs entirely from user space.

cachegrind is extremely useful, but unfortunately it seems that it's not updated with the current generation processors. Every time I run it, it complains about "unrecognized architecture" or something similar.

You should try htop and iotop as well. Although atop is generally better for more sysadmin-type things than htop is.

Not cool... "sudo perf record -a -g sleep 10" crashed my system badly (not even reisub would close it).

Are these tools (perf, atop, etc) useful and accurate in virtualized environments? For example, under Xen. Fine, I'll say it: on an EC2 or Linode instance. Would there be any skewing or clock issues due to virtualization?

Collecting the comments of HN readers with deep experience in this matter, on this thread, would be preferable to me trying to give a brief and uninformed Google search summary.

Some features of perf will work perfectly in an emulated environment. Perf's kernel tracing uses tracepoints in the kernel (built-in or dynamically patched in), and those will work just fine. However, perf's use of CPU performance counters will only handle those the virtual environment emulates; I don't know if Xen implements those.

the article mentions perf, valgrind, oprofile, etc. but misses the lttng tool

http://lttng.org

Is there a better viewer for lttng than the lttv gui? I got spoiled by QNX's amazing tracing infrastructure and analysis tools and long for something similar on Linux.

lttng is integrated into eclipse framework, the gui looks better than the gtk lttv gui

Do any of these run on MIPS? Profiling for embedded Linux can really suck sometimes...

lttng currently runs on: x86-32, x86-64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM

For Windows, xperf is in the systemwide-profiler category: http://msdn.microsoft.com/en-us/performance/cc825801.aspx

Is this like the "Sample" button in Activity Monitor on OS X?

its more like instruments and shark tools that you can find in xcode

Does anyone know how to install it on Arch?

maybe https://aur.archlinux.org/packages.php?ID=31439

Anyone know how to install this on CentOS?