FIO is also really useful in that its dizzying array of options will often prompt questions you didn't even think to ask about what you're trying to measure. Every time I am trying to design a storage benchmark for a certain purpose or trying to replicate a particular workload, I read the FIO documentation cover to cover to make sure I'm not forgetting any crucial details.

(Of course, FIO's options aren't quite exhaustive, but I can only think of two of three things where I've had to wrap or extend FIO to achieve what I needed.)