Didn't perf work? What difficulties did you encounter?

Used perf record for getting CPU efficiency numbers, and it worked very well, however to be able to reliably trace the code, I had to compile it in debug mode with no optimizations, and run with a single thread. That combination increased the execution time considerably.

Some of the Matrix and solver code coming from Eigen is heavily optimized for SIMD operations and using it with -O0 is just painful.

Moreover, I had to verify its memory sanity and used Valgrind for that. Using a full size problem meant it had to churn for days to finish execution.

Having deadlines doesn't always help in this stuff.

I found perf pretty crummy compared to vTune and even vTune is not fantastic at scale.

I think perf and valgrind is a fantastic combination. While valgrind has a big overhead, it can make a lot of things visible without instrumenting the code.

I also think Perf is phenomenal for its scope. It's showing performance metrics of the code without any processor dependency and instrumentation.