Sure it does, memory management impacts the performance of the task being achieved, writing an high-speed network driver.
By that measure, every project is a good measure of GC performance. Is that really a good argument?
I believe all general purpose languages let the code allocate memory and therefore will let you allocate memory in a way inefficient to your task.
The argument is that to eventually write Swift code whose performance matches the competition, one needs to work around ARC's implementation.
Hence why there is so much emphasis on value driven programming alongside protocols at WWDC talks.
Also, the Swift implementation is a bit questionable if performance is a goal. That is, why not try to remove the memory management from the inner loop? Probably the first thing to try is value types instead of reference types, which are more generally preferred anyway.