> the moment you hit something like that, it becomes a massive pain, and requires either "unsafe" (which is strictly more dangerous than even C, never mind Zig) or patterns like using indices instead of pointers

If you need to make everything in-house this is the experience. For the majority though, the moment you require those things you reach for a crate that solves those problems.