AFAIK, cargo doesn't have any feature to point out when a crate contains unsafe codeāso you pretty much need to grep the source of every crate you consume for "unsafe".
Auditing FFI is a whole other challenge, however :(
Not in binary libraries, hence why it is important to have a culture to only use unsafe if it really must be used.
You do have C libraries which you access through FFI. This is inevitably unsafe. We should be auditing more there. Though IMO it's still manageable, for most crates.
The issue you're talking about is related, but different.
In general when talking about safety in a language it's about the level of explicitness required to trigger unsafety.
I like the distinction made in the nomicon (https://doc.rust-lang.org/stable/nomicon/meet-safe-and-unsaf...) -- Rust comprises of two distinct languages. You have everyday Rust, which is completely memory safe, and "unsafe Rust", which looks similar to everyday Rust but is not safe. `unsafe {}` blocks are your FFI between the two. Looking at unsafe blocks as FFI is IMO a very useful mental model especially for understanding the changes to invariants involved.