I'm not a Ruby programmer, so I'm interested to hear your view on this: as a non-Rubyist reading Ruby code, the "assign variable to a new array" method is much more readable to me than than the .shift(n) method and is only 2% slower. Is shift(length-of-array) an idiomatic way to clear an array in Ruby? From my perspective the site looks like a great way to encourage confusing code for little gain.
No, it's just a benchmark testing many ways to clear an array; there's nothing idiomatic about shift(n) that I'm aware of. Benchmarking code should always be taken with a grain of salt, it can get pretty needlessly esoteric at times in the pursuit of speed.
I would be interested in if anyone knows how the garbage collector handles a.drop vs a = []. Maybe a.drop collects earlier, and the gc is causing the slowdown?
Totally agree that benchmarks should be taken with a grain of salt. I was scanning the ruby docs for Array and saw shift(n) so I thought it'd be fun to try it.
I'm pretty ignorant about how the GC affects these benchmarks - I'd like to investigate that further in the future.
GC should be disabled for proper benchmarks anyhow, honestly. Or at least measured. REE has handy switches to measure that sort of thing.
Disabling the GC means your benchmark is worthless.
1. Valid GC during execution: you are removing an actual factor;
2. "Invalid" i.e. unrelated GC during execution: your benchmark is too narrow in scope.
It depends on what you're benchmarking. If you're benching wall time, then yes, you need to have GC on, and you need to do a ridiculous number of iterations to average out GC runtimes. You also need to invoke the GC before you begin your benchmark; doing a collection pass early in the benchmark for all the objects created on startup/setup skews the benchmark.
Separate benchmarks should be conducted for execution speed (CPU time) and garbage generation. Saying "X is faster than Y" tends to suggest a CPU time benchmark. "X is faster than Y with W delta garbage and Z delta less time spent in GC" suggests a proper wall time benchmark.
On the other hand, 10k+ iterations likely isn't anything you normally experience, and may incur abnormal GC cost due to the sheer number. Though that's typically detectable by doing multiple sizes of tests.
In some cases the gain is more dramatic, and in other cases, that 2% gain may be worth it. If you're simply learning a language I don't believe looking at benchmarks is a good way to do so. Typically the most performant options are hacky at best (in most all languages), imo.
If I'm reading right, he's not surprised regez literals are faster than regex strings. He's surprised that regex search-and-replace is faster than a literal string search-and-replace.
Aaaah, that's a good point. That's quite a bit more rational.
You should use http://github.com/sandro/bench_press and share your findings :)
changed the example and sent a pull request. Thanks for pointing that out :)
It does convert string literals to regexps, which is kind of disappointing. I once did some benchmarking with Tcl, and there, glob patterns were significantly faster than regexps, let along string literals.
Nitpick on their array tests, however:
Ah, no. If o == false, that won't delete it. Also, you're calling object == true; how about a.delete_if {true}? Also:>gsubbing with string literals vs regex. For some reason regexp is faster. I suspect it converts string literals to a regexp anyway.
Really? You're surprised regex literals are faster than regex strings?
The gem however is interesting and nice and simple, I may end up using it for my future testing. Haven't looked through the code, however: anyone know if it's making any mistakes?