IMO this example should be taken with a grain of salt, since at a cursory glance it looks like they are leaning heavily on reference-types, which is not really idiomatic swift, and is a worst-case for tightly-looping code which this is. I would love to spend even a half hour trying to optimize this code to see what the results could be, but there is specialized hardware required to run the benchmark, so it's not really practical for 3rd parties to attempt to achieve a better result.
It's fair to say that ARC has performance issues, but the story is a bit more complex than that. By relying heavily on value types, it's possible to write Swift code which carries very little ARC cost (although there are additional issues with copy-on-write performance). Also ARC is more memory efficient than most GC solutions, so there are tradeoffs involved.
https://github.com/emmericp/ixy