> Isn't it the case that once you use unsafe even a single time, you lose all of Rust's nice guarantees

Inside that block, both yes and no. You have to enforce those nice guarantees yourself. Code that violates it will still crash.