News for hackers. Stuff that matters.
Re:Your nines are not my nines by hinkley on Tuesday July 16, 2019 @04:55PM (#20453804)
We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution?

If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding solution wrong to small-C n^2 issues (and a whole lot of space between for nlog(n) problems).

If I'm doing 95th percentile calculations that will not show up in my metrics. If I have a larger customer that's 10% of my traffic, almost half of their users could be having issues before my alerts go off.

And then there's explaining to your boss that 5 9's across twenty interacting services is around 99.98%, and that's only if degradation in one service doesn't cause failure in another.

Re:Your nines are not my nines by jameshart on Tuesday July 16, 2019 @07:34PM (#20454932)
The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small portion of your traffic.

Scatter plots and histograms are much better at telling you when the distribution of a stat has gone bimodal with a small but consistent group in the outlier group. Percentiles only make sense for telling you, when you already know what shape the distribution curve should be, how flat that curve is right now. They don’t tell you when the shape has changed.

Re:Your nines are not my nines by bosie on Tuesday July 16, 2019 @08:31PM (#20455316)
What scatter plot and/or histogram do you have in mind to dig up the small but consistent group in the outlier group?
Re:Your nines are not my nines by jameshart on Wednesday July 17, 2019 @08:39AM (#20458773)
Ah, yeah - figuring out what they have in common is the trick :)

Good APMs and trace tools will let you zero in on traces by characteristics - so if you notice there’s a bump in requests which have a 2 second load time, you can select them all and analyze how they are distributed - whether they are mostly one browser, one location, or one user even. But you need a solid strategy for tagging and logging traces.

Re:Your nines are not my nines by jgtrosh on Wednesday July 17, 2019 @04:26PM (#20463602)
You could statistically test whether your data matches the expected distribution with automated tests.
Re:Your nines are not my nines by samstokes on Tuesday July 16, 2019 @05:51PM (#20454234)
That's the "monitor from the customer's point of view" approach the OP alludes to. If you use tools like Honeycomb [1] that can easily and routinely answer questions like "show me the 95th percentile latencies for each of the 10 customers experiencing the worst latencies", then situations like you're describing are a lot easier to discover.

[1] https://honeycomb.io. Disclaimer: I used to work for them.

Re:Your nines are not my nines by deaddrop on Tuesday July 16, 2019 @06:57PM (#20454689)
>That's the "monitor from the customer's point of view" approach the OP alludes to.

...but now you're in a recursive problem: Who watches the watcher? If the watcher goes down, your insights are gone. Do you devote your entire engineering staff to monitoring, then?

A two-pronged approach would be better: Customer Touch-Point monitoring built into your product and external monitoring should your CTP monitoring go down. If your external monitoring goes down, you still have the CTP, so not all visibility is lost.

Re:Your nines are not my nines by mentat on Tuesday July 16, 2019 @06:05PM (#20454312)
Having standard shaped telemetry for SDKs (via OpenTelemetry hopefully now) then allowing opt-in aggregation of customer views, either as an observability product to peers or to the providers is what I've been trying to get interest in for the last few years. Having consistent data to show cloud support is also helpful, even if only for one company, especially if you can show you're usually right. A short blog post on this I wrote a bit ago: https://lightstep.com/blog/tough-conversations-with-cloud-pr...
Re:Your nines are not my nines by solipsism on Tuesday July 16, 2019 @07:36PM (#20454943)
What role does the word "shaped" have in your opening sentence? Telemetry that is standard-shaped? Standard telemetry that is shaped?

And what does it mean? And am I stupid, or is this not a term everyone knows?

Re:Your nines are not my nines by finnthehuman on Tuesday July 16, 2019 @07:26PM (#20454870)
>There are a million things that can go wrong that make this customer's experiences different from everyone else

In my experience the things that are easier to write-off as unique to one case, non-representative, or too rare to fix, so they don't have to be thoroughly addressed are warning signs of a robustness issue. Still doesn't mean that they'll get fixed then and there, but they often come around to bite you in the ass later.