But then those barriers are not for the refcount itself, but for the data the object contains. Different things.
On x86, stores are always ordered anyways, so fetch-and-add doesn't need to do any implicit memory barrier.
Barriers are never needed for accesses to a single location as all practical systems guarantee total ordering, that a given. But on a refcounted system the only place to put the barrier for correct visibility, it is together with the refcount update.
It is true that on x86 all stores and loads have the correct implicit barriers, but you still get a store-load (including its cycle cost) with lock xadd whether you want it or not.
Also many architectures with builtin fetch-and-add bundle it with an implicit memory barrier (that includes x86).