As a sysadmin. I don't love any other tool nearly as much as I love atop. It comes close to making all other interactive general-purpose performance monitoring tools redundant. The logging, too, is fantastic. There's nothing like looking through the graphs of my metrics gathered by collectd, wondering why a certain spike or pattern occurred, and being able to step through previous days per-process resource usage via atop and figure it out. There's a great introduction to it at https://lwn.net/Articles/387202/

Do you know of any manual or book-style treatment about using those tools for investigating the performance of Linux systems? Every once in a while I have to look at performance problems on Linux systems, but I always start from zero. I need a place where I can study the topic and refresh myself when necessary, because I'm apparently not capable of mastering it via occasional piecemeal experience.

The closest I've found (and spent 10s, if not hundred+ hours reading) are: System Performance Tuning, 2nd Edition (O'Reilly System Administration) by Gian-Paolo D. Musumeci [1] and the somewhat dated (and Solaris centric) Sun Performance and Tuning: Java and the Internet (2nd Edition) [2]

I'd love to hear that there were better book out there - if anyone has read either of those two, and has found a better one, I'll be one-clicking it in 30 seconds.

[1] http://www.amazon.com/System-Performance-Tuning-OReilly-Admi...

[2] http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/d...

Also don't forget Cachegrind. Cachegrind is amazing and runs entirely from user space.

cachegrind is extremely useful, but unfortunately it seems that it's not updated with the current generation processors. Every time I run it, it complains about "unrecognized architecture" or something similar.

You should try htop and iotop as well. Although atop is generally better for more sysadmin-type things than htop is.