If you add multi-threading and multi-core, there is lock contention for every single memory access to update the related counter.
Additionally, if there is a big data structure in use, reaching zero will trigger a cascade deletion events reaching stop-the-world like effect.
Which, depending on how the deletion was implemented, might trigger a stack overflow in very deep structures.
Herb Sutter has a very interesting talk about the bad issues of reference counting and how to work around them.
CppCon 2016: Herb Sutter “Leak-Freedom in C++... By Default.”
https://www.youtube.com/watch?v=JfmTagWcqoE
There are multiple ways to optimize reference counting, like deferred counting, cycle collectors and a couple of others, which when you are done with all them, ends up being quite similar to having a tracing GC anyway.
You only have to update the counter when creating (or destroying) new "owning" references that might outlive any other reference to the object. This is a rare operation, and doing it potentially from multiple threads in a way that requires atomic update is even rarer.
> Additionally, if there is a big data structure in use, reaching zero will trigger a cascade deletion events reaching stop-the-world like effect.
This is a side effect of deterministic disposal, of course. And it only happens when freeing up large object subgraphs, not just any and all "large data structures".
Yes, C++ implements reference counting in a non-ideal way. Swift is also quite bad. But one can do better.
Even a deletion cascade won't stop the world: only the single thread which reached zero will be "stopped". While in tracing GC, a "stop-the-world" stops all threads, even the ones which aren't doing any allocation or deallocation at the moment.
Also there are tracing GC that operate with local heaps.
In RC systems you don't escape latency spikes from cascading deallocations and nondeterministic malloc/free call latencies. The malloc impl underlying RC still needs to occasionally work quite a lot to recycle freed memory, just with additional constraints compared to non-RC GCs.
The downside is, of course, that it requires much more care (to avoid cycles)