It would be interesting to see how much wall clock time these applications run for. One answer to the question of how many iterations to run for is to compare it to the expected running time of real applications. If the expected running time is minutes, and the VM hasn't warmed up after ~1 minute, then it's pretty irrelevant whether it would "eventually" warm up or not. This would be one way to side step the halting-problem-esque question they've gotten themselves into with respect to how long to run the benchmarks for (which I'm pretty sure is intractable).