Yes shared_ptr is relatively expensive; avoid it in your inner loops, but know that it's there for the other 80% of your code.