With all that said, while this is a separate topic, I think that the complete lack of a spec for lots of details surrounding `unsafe` makes it still basically impossible to do correctly. I think Rust really needs some type of language spec, but it doesn't look like any work is being done on that front [0].
[0] https://github.com/rust-lang/rust/issues/30500 - I apologize for this link being a bit of a rabbit-hole, but I think the whole thing is a very interesting read. It continues on here (https://github.com/rust-lang/rfcs/issues/1447), and then to a repo for a Rust Memory Model specification here (https://github.com/nikomatsakis/rust-memory-model). Unfortunately, as you can probably tell, those conversions started in Dec 2015, and any real work on this front seems to have stalled almost a year ago.
This is true; what I'm getting at is that you shouldn't allow your unsafe code to depend on "largely the entire rest of the safe code."
Unsafe code should be used together with the visibility system, the borrow checker, and the trait system, so that it only depends on a very small amount of code within the same file or even the same function.
Then the vast majority of the program can be written in safe code that cannot violate the invariants of the unsafe code.