To be fair, std::shared_ptr is just a high-level component that's a part of the STL. Although it's defined in the C++ standard, it's just a generic shared pointer implementation designed to be as robust and bullet-proof as possible.

As with everything in the C++ STL, if you care about bleeding edge performance then you should be prepared to use more perform any (and less generic) components, whether it's data structure implementations or shared pointers.