I find it's the other way around. Quite often it's the developers who don't care if a menu is out of place or if an animation stutters.
The users, however, do see these things. They might not be able to articulate it, but they will see it. And call your app slow (animation stutters), unpolished (menu not rendered where it should) etc.
That's why, for example, MacOS has been getting so much flak these past years: they've stopped sweating the little details, and people noticed.
My sor of laypersons intuition about UX has been that it's just about reducing cognitive effort for new adopters. For things that don't really matter to functionality, make it consistent with what they'd expect given their context so they don't have to think about it. Hamburger menus aren't "intuitive," give them to a tech naive person and they'll be confused, but they sure feel that way today because of how everyone has been thoroughly trained in how to use them to navigate them, so why not take advantage of that social training to lower the adoption activation energy? I would think that I would rather people spend effort figuring out the parts that have to be different for the new functionality. I'm curious to hear from someone with actual design training if that's a reasonable approach, or if there might actually be some reason that the opposite is actually empirically true (like a unique UX makes people feel like power users and locks then into the platform or something?)
When I hear the other point that users dont care about UX what that boils down for me is they don't see all the little things that we see. They still expect a consistent user interface but seeing a menu not render exactly where it should, an animation that stutters for one frame or other small things. These are all things that we as developers scrutinize but the user doesn't notice all that often.
Now we should still scrutinize those little things. I feel like the thing that makes some apps feel more "magical" than others is how polished those little details are.