Copy constructor, destructor, move constructor, copy constructor.
All of them have to do counter booking, value count and accessors, also possibly handle weak_ptr booking.
Plus how it actually works isn't part of ISO C++, so you're betting your luck on how a specific C++ is actually going to implement it, and possible issues when compiler versions change, same compiler or to another one.
C++17 has copy elision as part of the standard now, so you can be guaranteed of the scenarios where C++ is required to outright skip the copy/move/destruction, leaving no code paths left for any refcounts. It’s not everything, there’s still plenty of optimizations on the table, but copy elision should eliminate a ton of refcount bookkeeping.
Aren’t refcounts in shared_ptr done as part of the copy constructor and destructor? It seems like vanilla copy elision would count as optimizing away the refcounts.