To Stallman, your counter-argument is a non-sequitur. His goal is maximizing user freedom. UX may be one dimension of user freedom, but at best it's secondary to certain prerequisite freedoms, such as the freedom to modify code so you can independently improve it, perhaps by improving UX for yourself and your community.
Many people sympathetic to Stallman, but more utilitarian, might argue that in the long term, user freedom is necessary for achieving optimal utility (e.g. UX, etc). And getting there might require short-term sacrifices in superficial aspects like UX.
With as many open and closed options that exist out there, competition is a thing and Open Source as a whole needs to compete on more than just being "the freedom option"
Stallman's valid points are entirely lost in the ocean-full of software like GNU Image Manipulation Tool that incite frustration from users.
Except that's exactly what human behavior does.
Very few people in the world, even most FOSS contributors and developers, share that belief that strongly. They might feel like FOSS programs give them certain advantages and rights, but morals? Stallman has mentioned that he would rather go homeless than run proprietary software. That level of devotion is exceedingly rare.
To approach someone, even a Linux user, and tell them that they need to give up their convenience for free software because what they are doing is immoral, that's a very hard sell. This is one reason GNU has mostly produced free software clones of better, commercial software, where practical concerns take a backseat to GPL.