Tell that to Linus:
Data structures that have visibility outside the single-threaded
environment they are created and destroyed in should always have
reference counts. In the kernel, garbage collection doesn't exist (and
outside the kernel garbage collection is slow and inefficient), which
means that you absolutely _have_ to reference count all your uses."
Just to balance your downsides of refcounting, some downsides of GC: requires halting the program periodically and trashing all the caches, which is inconvenient and sometimes impossible. And performance-wise it's unpredictable and can require lots of tuning, as Java programmers who work on allocation-heavy workloads can attest.
Garbage collection has a lot going for it too, but any garbage collecting algorithm will suffer from at least one of the problems you mention, in varying degrees. Reads and writes should not require memory management code, like updating a refcount? That rules out a large class of GC concurrent/generational algorithms that depend on read and write barriers. No extra memory on a per-object basis? This rules out many incremental mark-and-sweep algorithms that store the object color. (True, the color may be implemented with a side table, but so also may reference counts.) Play nicely with concurrent threads? This rules out a large class of stop-the-world collectors. Cache friendly? This rules out semispace copying collectors, which may even page in memory from disk during a collection.
I'd say the evidence shows that GC performs best when there's lots of memory available and low interactivity, so it's a good choice for server-type applications. For interactive applications with limited memory, especially mobile apps/games, GC is more problematic, and it's not so clear. I don't know if the GCs in XNA or Android are "halfway-decent", but they both have significant overhead that causes real performance issues.
And I’m well aware of modern GC implementations, but one of my goals is deterministic memory behaviour, a feature lacking in other functional languages. I swear I know what I’m doing. :)
[1]: Read: will be; the current implementation is interpreted but I’m working with a friend on an x86_64 VM target.
Refcounting is an extremely poor choice for memory management on modern machines. Even putting aside the issue that it can't handle cycles: constantly writing to memory to modify reference counts can be a huge performance hit (you often have to do it even when you are accessing objects in a read-only fashion). Also it requires extra memory on a per-object basis. Also it doesn't play nicely with concurrent threads. Also it's not cache friendly.
I'd expect any halfway-decent modern GC implementation to be significantly more efficient than reference counting.
About the only valid justification I can see nowadays for not using a GC is a hard-realtime latency requirement. Yep, GC pauses suck. But for efficient memory management in nontrivial systems, GCs are definitely the state of the art.