> The majority of C code doesn't manipulate pointers either

But it's not just pointers. C is unsafe at every turn. Assignments can give undefined behaviour, as can the arithmetic operators, as can varargs... the list goes on.

    int i;
    int j = i; // undefined behavior

    int m = 0;
    int n = 42 / m; // undefined behavior

    int x = INT_MIN;
    int y = -1;
    int z = x / y; // undefined behavior (assuming two's complement)
 
    printf("%d"); // undefined behaviour

> Rust is not going to magically solve these problems for unsafe code...

It doesn't need to. By providing a language where only a tiny fraction of a well-designed codebase needs to use the unsafe features, the number of safety-related bugs can be vastly reduced.

Rust does not aim for perfection. If you want that, your best bet is formal methods.