News for hackers. Stuff that matters.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by smanek on Saturday January 09, 2010 @02:24PM (#1041837)
I strongly disagree with Zed on this one (at least part of his diatribe - he makes some good points about sample size/ramp-up/etc).

I don't care what your mean or standard deviation are. If your performance isn't symmetrically distributed about the mean (and it probably isn't), standard deviation probably doesn't mean what you expect.

I only look at performance in 99th percentile (or sometimes, 99.9th percentile). If 99% of responses are being served in 100ms on my webapp, I'm happy. I don't care if the mean is 5ms with a 20ms standard deviation. As long as the performance at the 99th (or 99.9th) percentile meets a level of 'acceptability' that I have predetermined, I'm satisfied. And you should be too.

This is much easier to deal with than trying to do real math. Just make sure that no more than 1% (or 0.1%, as the case may be) of requests take longer than X.

Pre-emptive answer: I usually get someone asking me why I don't demand acceptable performance at the 99.99999th percentile or at the slowest request. It's simply an issue of resources. To get that kind of performance, you can't use garbage collection, context switches become an issue, disk buffering even comes into play. You have to basically write a hard real-time (or pseudo real-time) app, and that level of effort isn't worth it for a consumer facing webapp. If I was writing code for a pacemaker or something (and please, dear god, no one ever let me do that) it damn sure better be hard real-time though.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @04:21PM (#1042005)
I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example:

Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails).

Is that really an acceptable behaviour?

I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @04:45PM (#1042023)
Taking more than 100ms is not the same things as "time-out and die". Performance is focused on how long something takes and it normally assumes that the system is not broken. It would take a horribly broken system to timeout 1% of requests. Assuming that 1% of your requests taking long enough the user might notice, but still working is generally acceptable performance for a webapp. So a reasonable goal might be 99% within 150ms and 99.95% don't fail.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @04:56PM (#1042036)
smanek wrote: "I don't care what your mean or standard deviation are.", which if taken seriously, means exactly that 1/1000th request can take an hour to finish and he doesn't care. Anything over 10s in real-life means a time-out in connection, or user refreshing, or going to the next page, or...

Going to 99.95% not-failing connections doesn't help you much IMHO. It means that you've gone from every 16'th person having a problem to every 32'th person. It's not acceptable at all.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @05:49PM (#1042117)
Your also assuming that he / we measure per element not per page, which is just silly. Also, stating a minimum accepted value is not the same as ignoring the rest. It just places minimum reasonable performance that a context.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @06:24PM (#1042170)
I read his "Just make sure that no more than 1% [...] of requests" as actual requests. If it meant whole-page-load-time then fair enough.

"stating a minimum accepted value is not the same as ignoring the rest" - then what is it if you don't care about mean and stddev? If you do care about other requests you care about the mean (or stddev, or median, or whatever other parameter) but just don't put it in a scientific way. If you say reasonable, it might mean - it doesn't time-out. That means it's below X seconds. That means with the other maximum load time constraints, you expect a mean time for Y requests to be less than Z. Saying "Assuming that 1% of your requests taking long enough the user might notice" you just put a constraint on that parameter.

(just to explain why I keep arguing this point - because that's what the post is about (kind-of), people say "reasonable time", but don't want to set the actual mean/stddev - which is what they do care about)

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @07:02PM (#1042238)
No, because taking less than the threshold is effectively meaningless. So, when you calculate the mean and standard deviation the calculation is altered by meaningless trivia. When you add in the latency and rendering times 1ms and 30ms might as well be the same number. What you really want is to round everything under your thresholds up to that threshold and then look at the breakout as x% take 100ms or less, y% take 1 second or less, and z% take less than 3 seconds etc.

You then look at that breakout based on what the page is, or the time of day ect. But, from a performance standpoint the the only really important number is the first number, once enough of your site is fast enough having a few slow areas, or some issues at peak times is reasonable, and once it's slow enough for a user to notice it's a problem.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by alain94040 on Saturday January 09, 2010 @10:38PM (#1042577)
If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die

Your argument is mathematically correct, but from an engineering point of view, doesn't make sense.

Of course in theory a system which responds 99.9% of the time means that it's possible for 1 of 1000 requests to die horribly. But we are engineers. If I achieve 99.99% of my target, it means that it's highly likely that that one request will also behave with pretty good performance, but I can't be sure that it will be 100ms. It might be 150ms. Or 200ms. But as an engineer, I claim that I am not concerned and don't care about knowing precisely that number.

I think this is what the author of the original article misses completely: he is applying advanced math and refuses to consider that engineering tradeoffs are practical. Of course in theory they aren't. But in practice they are.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by kelnos on Sunday January 10, 2010 @07:35AM (#1042995)
You're arguing about the wrong thing: I still think the parent's position is reasonable and that's it's fine to only care about performance up to a certain percentile.

Maybe you don't like 99% and think it should be 99.99999%. Fine, doesn't matter. Do what makes you happy for your web app.

And besides, this "measurement" has nothing to do with the functional reliability of the app. 99.99% of requests completing within 100ms doesn't mean the other 0.01% of requests fail (as another poster noted); it just means they take longer. If you're worried about requests failing (as we all should be!), then that's a separate issue to deal with. So then you say something like, "99.99% of all requests must complete within 100ms, and no more than 0.0000001% of requests are allowed to fail on average." Or something like that.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by zedshaw on Saturday January 09, 2010 @06:44PM (#1042202)
That'll work, if you have access to 100% of the timings for 100% of the requests and can get that information without having your measurements also modify the results.

Otherwise, it's just easier to do samples and then collect meta-statistics (which are always normal). In that case you'll need to know error rates which require things like variance and std. deviation.

Another technique is to go with statistical process control theory which mostly rejects confidence intervals and instead focuses on live sampling of processes to watch for outliers that need justification. With those you're looking at std. deviation as a measurement of the range of what's been happening vs. what's currently happening.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by DaniFong on Saturday January 09, 2010 @10:39PM (#1042578)
Meta-statistics for website are typically normal?

I think for one way flow through systems this has a tendency to be true (and so is likely to arise on most test harnesses), but I recall that multi-way routing systems with feedback effects (like, say, IP networks or highways) tend to have substantially non-gaussian congestion statistics: typically worse that gaussian, actually.

Consider B. Huberman et. al.

www.hpl.hp.com/research/scl/papers/InternetCongestion/InternetCongestion.pdf

One problem with relying upon standard deviations when you are trying to design a reliable system is that the majority of the variance in many distributions is concentrated in rare events. This is particularly true with long-tails (e.g. a Pareto distribution with alpha <= 2). If you happen to have a system conforming nearly to such statistics, and you keep your eyes on the mean and the standard deviation alone, you'll end up severely underestimating the standard deviation (which might not even converge) and you could be designing a faulty system.

Of course, the real problem, how can one gain confidence in one's statistical analysis. Not so easily, really. So if there's two things people who know about statistics, it's that statistics is critical, and statistics is hard.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by gambling8nt on Sunday January 10, 2010 @01:14AM (#1042737)
What Zed is saying when he notes that meta-statistics are normal is that, thanks to the central limit theorem, the average and standard deviation of data sets collected from the same underlying probability distribution (with convergent average and standard deviation) will tend to be normally distributed (in the limit approaching infinite sample size), even if the underlying system behavior is far from a normal distribution. In practice you work with finite sample sizes, so an underlying distribution sufficiently far from normal will result in a non-normal distribution of meta-statistics--but in most applications, these sort of pathological distributions are largely irrelevant.

Take our example of looking at response time for loading a web page. There is some finite point (say, 10 sec) beyond which we no longer care how much longer it takes. So instead of considering the distribution of response times t, we consider the distribution of min(t, 10 sec). This distribution only has support over a finite interval, so its meta-statistics normalize rapidly as you increase the number of trials.

Using this will under-report the actual standard deviation in the response time (which might, as you say, not even converge), since we've eliminated extremely low probability events with very high response time, but as a practical matter this is largely irrelevant--if these events are high enough probability for us to care we'll notice them anyway. The point of this exercise is not to perfectly ascertain the underlying distribution of t, it is to develop useful predictions for system behavior in practice.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by DaniFong on Sunday January 10, 2010 @01:33PM (#1043412)
The calculated standard deviation from any finite sample size of a long tailed distribution (e.g. Pareto with alpha <= 2) will be off by a factor of infinity.The point is that, not only is the standard deviation irrecoverable in this case, but it's hardly the figure of merit if you do know it.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by gambling8nt on Sunday January 10, 2010 @01:56PM (#1043450)
Except that in real life there are no distributions with support outside of a finite interval in space or time; there's always some point when you stop running the system...if some packets don't arrive by that point, you generally don't care how much longer they would have taken.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by hadley on Sunday January 10, 2010 @09:44AM (#1043101)
Except that the sampling distribution of the standard deviation is a scaled chi-square, not a normal. The central limit theorem is only for the mean, not any statistic that you might dream up. It's trivial to think of many that would not converge even with a windsorised response time.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by gambling8nt on Sunday January 10, 2010 @11:16AM (#1043220)
Chi-squared distributions are well approximated by normal distributions close to the mean.

The point is not that arbitrary statistics will necessarily always be perfectly behaved (or even well behaved) on sampling data--it's that to make reasonably accurate predictions of system behavior, under certain practical conditions, these statistics are well-behaved, and an inexperienced statistician (as most people are) is less likely to make a gross error.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by DaniFong on Sunday January 10, 2010 @01:42PM (#1043427)
Practical conditions not including routing networks and the stock market, you may wish to add...
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by gambling8nt on Sunday January 10, 2010 @02:00PM (#1043457)
Real life situations have finite cutoffs in behavior that remove many pathological problems with certain statistical models.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by hadley on Sunday January 10, 2010 @09:49AM (#1043109)
The mean is asymptotically normal and the standard deviation is asymptotically scaled chi-square; the distribution of every "meta"-statistic will be different.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by tel on Saturday January 09, 2010 @07:15PM (#1042269)
Totally agree. This was obviously one of Zed's older, asshole dramatic posts and it doesn't read anywhere more clearly than him yelling about people failing to do statistics correctly and then blanket assuming normality on something pretty unlikely to be normal.

Your method is a safe bet as long as you can collect "enough" data to be sure that you're covering all cases you claim to be covering. That being said, power analysis and data interpretation (ramp up, confounders, etc.) are all more difficult and important than this rant conveys. One R function is not your cure-all.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by dspeyer on Sunday January 10, 2010 @05:14PM (#1043771)
This works if your request stream is homogeneous. If it isn't (and they usually aren't), you need to break it up by type and check the 99.9th percentile of each of them. Otherwise you could find that 99.9% of HTTP GETs are asynchronous AJAX polling and the requests that matter are slower.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Locke1689 on Saturday January 09, 2010 @03:23PM (#1041938)
I'm pretty sure your system is useless without a statistically significant sample size.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by jerf on Saturday January 09, 2010 @03:58PM (#1041973)
Sampling is not relevant to smanek's point, since he can look at all events. You don't need to sample when you have all the data.

On the other hand, I guess this argues against smanek's points being relevant to Zed's point, since statistics aren't really involved for the same reason. You don't need stats to analyze your performance if you have all the performance data; just read it off.

(Edit: I suppose his point that you shouldn't be using averages but just thresholds is still a statistical argument, against using the wrong tool like "average" on the grounds that you don't actually want the "average".)

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by lmkg on Saturday January 09, 2010 @07:12PM (#1042264)
Even if you have all the data of your current run, that's still a sample (broadly speaking) of all the possible runs in the future. This is one of the core aspects of statistics, but it's so fundamental that it gets glossed over rather than getting drilled into people's head like they need. As a result, I have to tell people that when running a survey, total responses are more important than response rate.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by thaumaturgy on Saturday January 09, 2010 @05:52PM (#1042127)
> I strongly disagree with Zed on this one...

I'm unsurprised.

Programmers have a remarkable belief that they are experts in every subject, and if you tell them that there is something wrong with the way they argue, they will argue with you about it.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by nailer on Saturday January 09, 2010 @06:26PM (#1042172)
This recalled to mind recently meeting a physicist who had some experience of programming at a party.

The physicist repeatedly insisted (and explained to me with an authoritative tone) that Python 'has no VM as it is a scripting language'.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by scott_s on Saturday January 09, 2010 @06:49PM (#1042207)
I'd rather discussions here avoid such broad and inflammatory generalizations.