When I talk about a higher level interface it may be something like this https://crates.io/crates/typed-generational-arena those sort of things are very common in Rust. In here the index is an usize, but you don't directly do math with it (also you have a generation number to guard against index reuse), so you never experience any `as usize` thing.
Of course that's not applicable to everything. Some domains will naturally require doing arithmetic with indices, which is annoying in Rust. Hopefully you enclose this into a library and try to forget the horrors (rather than using usize everywhere, which is objectively the wrong thing to do)
Then you have libraries like petgraph that does use u32 indices by default, to save memory https://docs.rs/petgraph/latest/petgraph/graph/type.DefaultI...