That said, economics doesn't rest on this. Macroeconomics doesn't care, nor micro/labor/health/sports/developmental econ either.
Sure trying to predict how someone (and more interestingly how groups) will behave based on their psych profile is an important area of research, but the aforementioned subfields of econ already have well working assumptions about how people will behave in the aggregate, even if they can't derive it from some exact utility function.
There's nothing in the rational agent model that assumes that everyone has the same definition of utility function. Different people want different things.
Of course, it's fine as a very-very general fundamental theory, if you then want to study how people's revealed and non-revealed (old names for implicit and explicit) preferences aggregate into a utility function. (There's a whole bunch of math about pairwise comparison matrices. Lately there's some movement in that space about using perturbation to model inconsistencies in preferences, etc.)
What even the general theory is useful for, is to quickly reduce the question of the very general behavior/decision problem in to "show me your utility function and I'll tell you what you will do". Of course then that becomes a problem. How to model utility, how to formalize expected utility over all possible actions. What about priors (as in path-dependence, so do we want to encode that into the utility function - and constantly dynamically update the function, like a Bayesian agent would update its beliefs - or somehow manage it separately)?
...
Basically I was trying to persuade people to don't say things like "economists are dumb, because people are not rational utility maximizers", because that's a false implication. Of course they are, but - just like you said - this doesn't help us better predict people's (agents') behavior, just gives us a new task to model their utility functions. And that's where regret comes in. (Which is basically risk-aversion, which is what behavioral economics studies. People are not symmetric about positive and negative expectations. They value avoiding negative-utility events more than they value encountering positive-utility events. And of course with this insight now we can build better economic models - for example we can use this to "explain" why wages are more sticky - especially downward - than we would expect, also with some handwaving we can explain why in times of crisis people are let go instead of decreasing their hourly compensation, why some people are so gullible [peer pressure, confirmation bias ~ cognitive dissonance], and so on.)
Even when entities (corporations, for example) are trying to maximize utility, and an optimal decision is desired, there are issues with how much time and resources can be spent making decisions, so optimality has to be bounded in various ways (do you wait for more information? Do you spend more time and compute on calculating what would be optimal? etc.).
I keep seeing something like this in workplaces people don't join forces they avoid friction. Until something (crisis) or someone (capable leader) flips the thing on its head.
There has been quite a lot of study in economics about cooperation. My favorite is Eleanor Ostrom's work on "the commons". She observes that with a certain set of rules, discovered across the world and across varying geographies, people do seem to be able to cooperate to maintain a natural resource like a fishery or a forest or irrigation canals for hundreds of years. Her rules are here (https://en.wikipedia.org/wiki/Elinor_Ostrom#Design_principle...).