> you need to be unsafe if working with memory mapped structs

Perhaps when defining their address and layout, or if they are a hardware structure that affects memory like a page table or TLB or a DMA controller, but beyond that, why? Or is that just the main use case you're describing?