Two extra memory accesses plus whatever overhead for you have for locking if you are multi-threaded? That's pretty much the opposite of fast.

With refcounting you have:

  - Slow memory allocations (need to manage a fragmented heap)
  - Slow accesses and pointer handovers (updates to the refcounter)
  - Slow free (need to manage the free list)
  - No asynchronous pauses (since there is no garbage collector)

With a GC, you get:

  - Fast allocations (usually just an "add" instruction since the heap is not fragmented)
  - Zero cost accesses and pointer handovers
  - Zero cost free (just stop using the pointer)
  - Some asynchronous pauses and CPU usage while running the GC

It turns out that the cost of the first three points when using refcounting are much higher than that of the GC. Another reply to your post included references to actual research on this subject.