Making them raw pointers is an enormous amount of work, and "delete them when you know it is safe" very likely is harder than finding the misuse of the smart pointers.

> I have to do mumbo-jambo dance to make it shift gears as I wanted.

You do not have to do that.