Most games don't do any heap allocations at al during runtime(or at least very very few), so you shouldn't/don't end up with garbage collector pauses during normal gameplay. If you refcount your main structures, they can be auto cleaned up /removed when changing levels/loading scenes/swapping chunks out on the fly.
FWIW I wouldn't use reference counting in game development if I could avoid it, because it has pretty bad performance characteristics as well (the only thing it really has over GC is that it's deterministic, and that, with the exception of in Objective C, you can choose where and where not to use it).