Users cannot tell you what's wrong with your design. They don't know. It "feels wrong" or they "don't like it". But they CAN tell you that your design is BAD. It's up to you (or other designers) to tell you what's wrong with it. But if the users don't like it, you failed, and it doesn't matter if other designers think it's good.

Your target is the users, if the users aren't having a good experience, you're not doing it right.