Totally agree that benchmarks should be taken with a grain of salt. I was scanning the ruby docs for Array and saw shift(n) so I thought it'd be fun to try it.
I'm pretty ignorant about how the GC affects these benchmarks - I'd like to investigate that further in the future.
GC should be disabled for proper benchmarks anyhow, honestly. Or at least measured. REE has handy switches to measure that sort of thing.
Disabling the GC means your benchmark is worthless.
1. Valid GC during execution: you are removing an actual factor;
2. "Invalid" i.e. unrelated GC during execution: your benchmark is too narrow in scope.
It depends on what you're benchmarking. If you're benching wall time, then yes, you need to have GC on, and you need to do a ridiculous number of iterations to average out GC runtimes. You also need to invoke the GC before you begin your benchmark; doing a collection pass early in the benchmark for all the objects created on startup/setup skews the benchmark.
Separate benchmarks should be conducted for execution speed (CPU time) and garbage generation. Saying "X is faster than Y" tends to suggest a CPU time benchmark. "X is faster than Y with W delta garbage and Z delta less time spent in GC" suggests a proper wall time benchmark.
I would be interested in if anyone knows how the garbage collector handles a.drop vs a = []. Maybe a.drop collects earlier, and the gc is causing the slowdown?