One counter-example is Mathematica. I can mutate any object any time I like, and doing so is extremely common. Yet under "pointer" its help tells me about some unicode symbols, and some GUI stuff about mouse pointers.
In reality, some of its lists are dense memory like C, some contain arbitrary objects (such as expressions, or high-precision numbers), but these implementation details aren't part of the language, and most users never know them.
0-based arrays are based on hardware implementations; 1-based arrays are based on abstract collections; you could make the argument that 0-based arrays are a leaky abstraction, but the question then becomes: what abstraction? C explicitly tried not to abstract away the underlying hardware.
Surely the point of any language at all is to abstract away some details of the underlying hardware. Both for portability and to make things easier for human brains. How much you should abstract away obviously varies, based on what you're trying to do, and on how varied the hardware is.
And it depends on how much computation & memory you can give your compiler. One reason new languages can have nicer abstractions is that they don't have to compile on 1970s hardware.
In Mathematica's case (and most other languages, sadly), presumably the concept of 'variable' is overloaded with the semantics of a pointer/reference. Meaning, you cause and observe changes to locations via a variable.
Any language that makes use of mutation for primitive constructs (not saying that's the case here specifically) needs an understanding of "observing/causing change" pretty early on.