I wonder if pointer arithmetic specifically is the sticking point here; when using pointers in other languages I don't ever do pointer arithmetic on them, so the integral value of the pointer is easily ignored.

For those wondering, given the context of Rust here, both references and C-style pointers are first-class in Rust, but references (which are overwhelmingly more common) don't directly permit pointer arithmetic at all, and would require one to first cast the pointer to an integer.