Scatter plots and histograms are much better at telling you when the distribution of a stat has gone bimodal with a small but consistent group in the outlier group. Percentiles only make sense for telling you, when you already know what shape the distribution curve should be, how flat that curve is right now. They don’t tell you when the shape has changed.
Good APMs and trace tools will let you zero in on traces by characteristics - so if you notice there’s a bump in requests which have a 2 second load time, you can select them all and analyze how they are distributed - whether they are mostly one browser, one location, or one user even. But you need a solid strategy for tagging and logging traces.
[1] https://honeycomb.io. Disclaimer: I used to work for them.
...but now you're in a recursive problem: Who watches the watcher? If the watcher goes down, your insights are gone. Do you devote your entire engineering staff to monitoring, then?
A two-pronged approach would be better: Customer Touch-Point monitoring built into your product and external monitoring should your CTP monitoring go down. If your external monitoring goes down, you still have the CTP, so not all visibility is lost.
And what does it mean? And am I stupid, or is this not a term everyone knows?
In my experience the things that are easier to write-off as unique to one case, non-representative, or too rare to fix, so they don't have to be thoroughly addressed are warning signs of a robustness issue. Still doesn't mean that they'll get fixed then and there, but they often come around to bite you in the ass later.
If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding solution wrong to small-C n^2 issues (and a whole lot of space between for nlog(n) problems).
If I'm doing 95th percentile calculations that will not show up in my metrics. If I have a larger customer that's 10% of my traffic, almost half of their users could be having issues before my alerts go off.
And then there's explaining to your boss that 5 9's across twenty interacting services is around 99.98%, and that's only if degradation in one service doesn't cause failure in another.