People do it all the time. For example, in C++ with modern owning pointers like std::unique_ptr<> and std::shared_ptr<> it's really not that bad.