News for hackers. Stuff that matters.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @04:21PM (#1042005)
I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example:

Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails).

Is that really an acceptable behaviour?

I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @04:45PM (#1042023)
Taking more than 100ms is not the same things as "time-out and die". Performance is focused on how long something takes and it normally assumes that the system is not broken. It would take a horribly broken system to timeout 1% of requests. Assuming that 1% of your requests taking long enough the user might notice, but still working is generally acceptable performance for a webapp. So a reasonable goal might be 99% within 150ms and 99.95% don't fail.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @04:56PM (#1042036)
smanek wrote: "I don't care what your mean or standard deviation are.", which if taken seriously, means exactly that 1/1000th request can take an hour to finish and he doesn't care. Anything over 10s in real-life means a time-out in connection, or user refreshing, or going to the next page, or...

Going to 99.95% not-failing connections doesn't help you much IMHO. It means that you've gone from every 16'th person having a problem to every 32'th person. It's not acceptable at all.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @05:49PM (#1042117)
Your also assuming that he / we measure per element not per page, which is just silly. Also, stating a minimum accepted value is not the same as ignoring the rest. It just places minimum reasonable performance that a context.
Re:Programmers Need To Learn Statistics Or I Will Kill Them All by viraptor on Saturday January 09, 2010 @06:24PM (#1042170)
I read his "Just make sure that no more than 1% [...] of requests" as actual requests. If it meant whole-page-load-time then fair enough.

"stating a minimum accepted value is not the same as ignoring the rest" - then what is it if you don't care about mean and stddev? If you do care about other requests you care about the mean (or stddev, or median, or whatever other parameter) but just don't put it in a scientific way. If you say reasonable, it might mean - it doesn't time-out. That means it's below X seconds. That means with the other maximum load time constraints, you expect a mean time for Y requests to be less than Z. Saying "Assuming that 1% of your requests taking long enough the user might notice" you just put a constraint on that parameter.

(just to explain why I keep arguing this point - because that's what the post is about (kind-of), people say "reasonable time", but don't want to set the actual mean/stddev - which is what they do care about)

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by Retric on Saturday January 09, 2010 @07:02PM (#1042238)
No, because taking less than the threshold is effectively meaningless. So, when you calculate the mean and standard deviation the calculation is altered by meaningless trivia. When you add in the latency and rendering times 1ms and 30ms might as well be the same number. What you really want is to round everything under your thresholds up to that threshold and then look at the breakout as x% take 100ms or less, y% take 1 second or less, and z% take less than 3 seconds etc.

You then look at that breakout based on what the page is, or the time of day ect. But, from a performance standpoint the the only really important number is the first number, once enough of your site is fast enough having a few slow areas, or some issues at peak times is reasonable, and once it's slow enough for a user to notice it's a problem.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by alain94040 on Saturday January 09, 2010 @10:38PM (#1042577)
If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die

Your argument is mathematically correct, but from an engineering point of view, doesn't make sense.

Of course in theory a system which responds 99.9% of the time means that it's possible for 1 of 1000 requests to die horribly. But we are engineers. If I achieve 99.99% of my target, it means that it's highly likely that that one request will also behave with pretty good performance, but I can't be sure that it will be 100ms. It might be 150ms. Or 200ms. But as an engineer, I claim that I am not concerned and don't care about knowing precisely that number.

I think this is what the author of the original article misses completely: he is applying advanced math and refuses to consider that engineering tradeoffs are practical. Of course in theory they aren't. But in practice they are.

Re:Programmers Need To Learn Statistics Or I Will Kill Them All by kelnos on Sunday January 10, 2010 @07:35AM (#1042995)
You're arguing about the wrong thing: I still think the parent's position is reasonable and that's it's fine to only care about performance up to a certain percentile.

Maybe you don't like 99% and think it should be 99.99999%. Fine, doesn't matter. Do what makes you happy for your web app.

And besides, this "measurement" has nothing to do with the functional reliability of the app. 99.99% of requests completing within 100ms doesn't mean the other 0.01% of requests fail (as another poster noted); it just means they take longer. If you're worried about requests failing (as we all should be!), then that's a separate issue to deal with. So then you say something like, "99.99% of all requests must complete within 100ms, and no more than 0.0000001% of requests are allowed to fail on average." Or something like that.