News for hackers. Stuff that matters.
Re:Unladen Swallow can now compile all Python code to nativ... by tl on Wednesday July 15, 2009 @01:06AM (#705172)
Shouldn't there be max along with min and avg? Std dev is higher in many cases. Could there be an issue where max (worst case) has worse performance in Unladen Swallow?
Re:Unladen Swallow can now compile all Python code to nativ... by kragen on Wednesday July 15, 2009 @02:38AM (#705275)
I don't know if this is the case here, but often in benchmarking, "min" is the real answer of how fast something is going, and any time on top of that is some kind of overhead that you weren't trying to measure — some other process was taking up CPU time, you had to page in part of libc, crap like that. "avg" serves to tell you whether that overhead was reasonably small, or whether you need to run your tests again (on a quieter machine, say) to get a smaller "min".

This depends on the assumption that what you're measuring is actually a deterministic process. There are cases where it's not; as an example, the Self papers have all kinds of moaning about the measurement imprecision caused by physically-mapped direct-mapped caches without OS page coloring, but with more than one page of cache, on old SPARCs. For any particular logical→physical mapping, your measurements would be reproducible, but when it changed (if you got paged out and back in, or if you restarted the virtual machine in a new process) you would get different results.

Re:Unladen Swallow can now compile all Python code to nativ... by imbaczek on Wednesday July 15, 2009 @12:07PM (#705933)
max is useless in benchmarking; there's no upper limit on how long any computation can take.
Re:Unladen Swallow can now compile all Python code to nativ... by kragen on Wednesday July 15, 2009 @07:16PM (#706822)
If that were true, real-time software would be impossible. (Unless you're talking about the risk of your hardware malfunctioning, I guess.) Practically speaking, max can be useful.