It depends on what you're benchmarking. If you're benching wall time, then yes, you need to have GC on, and you need to do a ridiculous number of iterations to average out GC runtimes. You also need to invoke the GC before you begin your benchmark; doing a collection pass early in the benchmark for all the objects created on startup/setup skews the benchmark.

Separate benchmarks should be conducted for execution speed (CPU time) and garbage generation. Saying "X is faster than Y" tends to suggest a CPU time benchmark. "X is faster than Y with W delta garbage and Z delta less time spent in GC" suggests a proper wall time benchmark.