These issues would be a lot less troublesome if hardware offered basic security, like zero-cost poison-on-overflow and bounds-checked pointers, either of which would solve any risks of just letting it be. But right now there's the pragmatic question of not only whether this is a bug you could hit accidentally, but whether it's a bug you could exploit intentionally to corrupt memory, so it's better to just do the technically-correct thing even if it seems pointless.
It's less concerning for C#, which is a managed language, of course.
As soon as your array is storing objects that are 2 bytes or larger, you can't overflow the unsigned integer any more. The same applies for a 64 bit address and 64 bit integers. Never mind that most 64 bit hardware in existence is actually limited to a 48 bit address space, in which case the bug CAN NOT be exercised even with signed integers.