I doubt you'd have the sort of trouble with pointers and unsafe blocks that you suggest. Most typical pointer patterns don't even require lifetime annotations, let alone bypassing the borrow checker.

I also think the "convenience features" are a pretty big deal. Generics with traits are a pretty big value add for their complexity, and to me affine types take away a lot of the complexity of C.