It doesn't "break"...you just have to think about what you're doing. Which, again, is the point.

If some external code is passing you pointers, then using shared_ptr is exactly the wrong thing to do, because you (presumably) don't own the memory. And if you do own the memory, there's no problem putting it in a shared_ptr. The fact that you can't blindly stuff everything into a shared_ptr is a feature, not a bug.