You'd be surprised how well this feature works. I don't recall ever having had an issue like the one you describe. The browser isn't changing the entire page layout, mind you. It only re-layouts the text when all you're seeing on screen is pretty much a single div. I guess you have to experience it to understand.
That is not something I would want in many cases where I'm zooming in on mobile, if I understand you. A lot of time I'm doing that so I can accurately click a link that was too small. I want the text to stay positioned so I can zoom to a specific part of it without it changing underneath me. That's what a zoomed in view means, IMO, especially when considering the origin of the meaning.