In the company I work for, we call it "customer experience." We have an entire team of people who work face-to-face with actual customers, observing how they interact with our products and services to improve things.
It's why we have a 96% satisfaction rating in an industry where 40% is considered off the charts good.
But this will never fly in the tech sphere because it requires actual feet on the ground and talking to wetware. Some middle manager along the way would say, "it doesn't scale" and the project would get shelved.
However, you're right that it's too rare. The number one problem I've seen is that executives are convinced that they already know what users need. And when they get real data disproving their hunches, they really don't like it.
Remember when Google was still small and functional enough to produce excellent software with a good focus on design consistently? I do and I miss them. But all good things fade over time.
Clearly Google haven't architected a system that can put (or attract) the right talent in control of the right places. Small fast growing firms naturally attract talent and "game recognizes game" usually so it was easier for them back in the day.
It's just interesting to think of all the firms with a legitimate pool of great talent simply not executing properly (Magic Leap is the perfect example of this). It's a sad waste of resources which is heavily responsible for the people running things (IMO).
Ultimately, we make things that serve other people. All of us need to pay attention to how our work impacts the users we serve. Experts can help with that, but the Lone Design Ranger isn't nearly as powerful as a cross-functional, user-focused team.
I would also suggest that at least in tech, smaller firms aren't the problem here. As you say yourself, the small, neighborhood grocery store can get this stuff right just by paying attention. I think the big problems come at scale, where managerialism and over-focused metrics can easily disempower people and get them focused on the wrong thing.