Swift does not have a GC managed runtime in the same sense as the other languages you listed [0], it’s basically a bunch of refcounts inserted by the compiler, with similar performance characteristics to Arc in Rust or shared_ptr in C++. (For classes, at least. Structs are value types and stack-allocated.)
[0] Yes, automatic refcounting is a form of garbage collection, but Java/C#/Go use tracing GC’s and not direct refcounting, whereas with Swift it’s more like the compiler is wrapping all objects in a shared_ptr for you, and so destruction is explicit and happens at exact points.
When a GC-managed language uses reference counting as implementation algorith, the compiler might be able to optimize the reference counting to only occur when it is unavoidable or too costly to reason about, in a similar vein to bounds checking.
When using library types for reference counting, there is no way for the compiler to implement such optimizations, unless the types are somehow tagged with compiler intrisics so that they could apply the same kind of optimizations.
Aren’t refcounts in shared_ptr done as part of the copy constructor and destructor? It seems like vanilla copy elision would count as optimizing away the refcounts.
All of them have to do counter booking, value count and accessors, also possibly handle weak_ptr booking.
Plus how it actually works isn't part of ISO C++, so you're betting your luck on how a specific C++ is actually going to implement it, and possible issues when compiler versions change, same compiler or to another one.
Languages with GC-managed runtimes (Java, C#, Go, Swift, et. al.) are actually significantly more performant for almost all heap-bound use cases, actually. Reference counting kinda sucks for typical code.