Swift is distinctly not Rust although it looks a lot like it on the surface. It uses reference counting, just like Perl, Ruby or Python. RC is the most rudimentary form of garbage collection. Also has its problems, like dealing with circular references.

And yes, CL is fully GC. But my point is that's not the culprit. If your problem calls for allocation and management of heap memory, it has to be done one way or another, manually or via GC. GC overhead in that light can really be negligible, and it's other concerns (like scheduling and real time constraints) that typically make its use problematic.