I doubt anyone reasonable even claims that a perfect society is possible - someone will always feel wronged regardless of their circumstance. I don't think it's unfair to say that the main function of a society is to reduce the likelihood that a person of said society gets wronged. Which brings the question of how does a society do that without enforcement of laws? If we accept the NAP is a law, who reprimands those who break it? Who actually defines what it is in the first place? Undoubtedly there are many different interpretations of it and it's not possible to achieve a feeling of fairness in a society if not everyone agrees on its basic structure and laws. If you feel like these are too basic of questions, feel free to link me to what you would consider a definitive explanation.
For me, it all basically reduces to the idea that Bastiat articulated, that "government is just the collective extension to our individual right to self defense". So even as a radical individualist anarchist / anarcho-capitalist, I can still see a role for something like "government", but only if it's use of force is restricted to defensive use, and it respects the idea that "if an individual doesn't have permission to do something, then multiple individuals don't have permission to either".
IOW, voluntarily organized collectives, assembled for mutual defense, OR individuals acting in self-defense, are how the NAP would be "enforced".
But again, there's no "perfect" political system, IMO. Conflicts could still arise, but I don't think institutionalizing the use of force is a good approach.
I don't know any libertarians or ancaps who insist that the non-aggression principle solves anything, much less everything. What we say is that it's simply the ethically Right Thing To Do, and that if "solving" a problem requires violating the NAF, then it's better left unsolved, even if there are negative consequences.
The idea that libertarians believe in some kind of "libertopia" is one of the most persistent, while completely unfounded and mistaken, myths that exists about libertarians. It's the libertarians who, more than any other group, take reality into account by acknowledging that we can't engineer a "perfect" society by using force against one another.
Libertarianism has pathologies at the edges? Big deal, so does every system that has ever been proposed as an alternative.