The thing is, you're going to be wrapping in a class anyways, so why not move all the nonsense into a normal destructor and then use normal smart pointers?

I suggest that this is somewhat solving the wrong problem.