> and only if you're very lucky the compiler will condense this down to the same few dozen lines of actually needed code

The code consists almost entirely of templates and inline functions and every modern C++ compiler will optimize that away. In a release build there is basically no overhead compared to using raw pointers.