Measure code execution time accurately in Python

Re:Measure code execution time accurately in Python by colmmacc on Saturday January 18, 2020 @01:24PM (#22086012)

There's another reason too; there are strong reasons why the timing of successive code executions should not be linear. Typically the first call to a function will incur the cost of fetching data into a cache. At minimum the basic blocks for the function will be loaded through the l1/l2/l3 pipelines.

2, 3, 4 function calls in a row will benefit from that caching, but also help train the speculative execution engine. At some point that speculation may radically alter performance. Later, with a sufficient number of runs, the timing will also suffer from preemption, context-switches, or other interruptions that can invalidate both the cache and the speculative execution engine.

The idea that there is a fixed performance time for a function is naive on modern hardware.

[ Parent | on HN ]

Re:Measure code execution time accurately in Python by cscheid on Saturday January 18, 2020 @01:31PM (#22086071)

You make a good point I hadn't considered. In that case, not even robust least squares will save you, because the samples are not independent of one another. You'd have to start looking into explicit time-dependent methods and yikes, I have no idea of what the literature says about time dependence and outliers. I wouldn't be surprised if it's "here be dragons" territory.

[ Parent | on HN ]

Re:Measure code execution time accurately in Python by gdxhyrd on Saturday January 18, 2020 @02:06PM (#22086338)

You are speaking as a stats person. :)

While there are dozens of things that can go south in benchmarking, how to solve them is not about applying advanced stats, but about understanding and eliminating the sources of noise.

Most benchmarks in CS can be done quite easily as long as one understands all the technology stack.

[ Parent | on HN ]

Re:Measure code execution time accurately in Python by cscheid on Saturday January 18, 2020 @02:50PM (#22086652)

I'm not a stats person! But I have been burned in real projects by doing the wrong thing. I just have learned --- the hard way --- to honestly appreciate it.

[ Parent | on HN ]

Re:Measure code execution time accurately in Python by colmmacc on Saturday January 18, 2020 @01:59PM (#22086278)

Speculative execution optimizations, e.g. branch prediction, are even data dependent. It's a very complex measurement space!

[ Parent | on HN ]

Re:Measure code execution time accurately in Python by jwkane on Saturday January 18, 2020 @01:36PM (#22086102)

"The idea that there is a fixed performance time for a function is naive on modern hardware." This is exactly it, though it can be further compounded by runtime optimization in software.

[ Parent | on HN ]