We used to track percentiles but discovered Asanas struggled to empathize with them. We switched to defining semantic performance bands (eg < 100ms, < 300ms, < 1s, < 3s, < 5s, < 10s). For each action, we track the percentage per band. Asanas seemed to empathize better with time as the independent variable and percentage as the dependent variable.

I would also suggest that you do the same with a metric for "slowest action in 15 minute period of user input". 99% of actions are fast but you have hundreds of actions over 15 minutes, people will notice those slow ones.

Thanks for the advice! We believe that users remember the slowest actions as well. Your strategy sounds like it will help identify the long tail of rare yet slow actions.

Here is another tip to track down that long tail.

"Tracer bullets". Have a flag that can be put on an action that causes it to be logged very verbosely. In a way that travels through your system. And then aggregate those logs, process them, and make them available. Then randomly tag a small fraction of actions with that flag. Like 0.1%. Because you do this relatively rarely, you can make verbose logs without overwhelming your system.

When you receive a complaint about foo occasionally being slow your first stop is to see if you've got a concrete action that was slow, look at the picture from the logs, and try to figure out why. This will greatly help in reproducing rare problems or, if you can't, at least identifying where they come from deep in your system.

Can you elaborate on what an Asana is and what "empathize" means? I'm not an Asana user, so perhaps I'm unfamiliar with the jargon.

By Asana, I just meant employee. More people intuitively understood our metrics when we switched our framing.

Another benefit of switching that framing is that it makes aggregation easier.