While having some support for garbage collection is good, reference counting is not is a rather expensive way to implement that for applications. This becomes especially bad in multicores since it may dramatically increase the number of writes to shared object cache lines.
Wouldn't a lot of that be mitigated, though, by using tagged pointers in place of RC structs where possible? Seems like an obvious optimization.
Not really sure what the tag in the pointer would be used for. Could you give an example.
In general, reference counting has the problem that it needs to update the reference count. If you have a read-only data-structure these updates to the references will introduce writes that may severely impact performance since a write introduces cache consistency communication, while reads are communication-free.
FWIW, Swift is categorized as a systems language in the opening pages. But, then, so does Go in its FAQ. To Swift's credit, at least it has deterministic memory management through ARC.