A key thing to consider is that there's nothing innately unsafe about casting a pointer to a number. It's when you go the other way around that problems crop up.

As long as dereferencing a raw pointer is considered unsafe, you're fine. Casting it has no actual effect at the machine level.