Linguist here. There’s no right way to understand your own language. The language is what native speakers collectively think that the language is. If they (by the way, am I referring to op now or to the native speakers?) think that something sounds unnatural, then it is unnatural to them. You can’t be wrong about this, just as you can’t be wrong about whether it feels natural to hug someone you met five minutes ago.

Everything we as linguists can do is to say that x feels natural to this many people and unnatural to this many people. However, when writing something, if enough people think that something feels unnatural, maybe you should investigate if you can phrase things differently so that more people will accept it.

With regard to your question, it works either way so there's no particular ambiguity. But since pronouns usually go to the most recent appropriate person/group, and you had not yet referred to the OP explicitly, the "they" could refer to a subset of native speakers who find it unnatural which also happens to include the OP, or just to the OP given the broader context of the message. And either way your comment comes out meaning the same thing:

People feel what they feel and can't be wrong about what they feel (because it's feeling, not a rational thing).

I don't disagree with that in any way whatsoever. But it doesn't mean that the OP is fundamentally correct either. I find the phrase "he/she" to be awkward, ignoring whether it's inclusive or not (which it isn't). "He" is fundamentally gendered despite what many people want to believe and is even less inclusive (even if we assume only the male and female genders, it necessarily excludes half the population). Consequently, singular "they" ends up being the most natural way to express a non-gendered arbitrary person, especially given that English has an 800+ year tradition of using it in that sense.

Also linguist here. You're right that there's no right way to understand your language, but it's also the case that you need to take a speaker's self-report with a grain of salt.

That speaker has probably seen singular they countless times without raising an eyebrow, and probably used it themselves. They may report being uncomfortable with it when their attention is called to it, but that doesn't always reflect their actual usage.

You get a better understanding by analyzing a corpus than by asking people. And the corpus makes clear that a lot of people have been comfortable with singular they, for a long time. I've even seen examples of people decrying singular they in sentences using singular they.

The self-reports are also data that's informative to the linguist. But they need context to gain perspective, because they're not always identical to what the speaker generates or accepts.