Quakers, hippies, the anti-war movement, and many others would disagree with you.

Elaborate? If China starts dropping bombs in the US, do they think that we should not retaliate?

That's the idea. Quakers will never take up arms. In both world wars they would actively help in peaceful ways (helping widows, sending care packages to POWs, etc), but their actions remain peaceful.

>Early Quakers were among the leaders of the anti-slavery movement in the United States. Quakers are also pacifists, responding to wartime calls for service seeking out non-combat roles such driving ambulances or serving in conscientious objector work camps.[0]

[0] https://encyclopedia.ushmm.org/content/en/article/quakers https://en.wikipedia.org/wiki/Peace_Testimony

This would really only work in the US, where invasions are very rare. Also, there's a difference between refusing to fight on foreign soil (like in ww1/ww2), and defending your homeland. eg. if the enemy is right outside their village, and they're going to rape/kill/pillage them, are they not going to fight back?

There have been similar cultures throughout history and throughout the world. Many Native American or Pacific Islander tribes refused to take up arms. Those tribes are no longer with us, but the cultural idea that one should not inflict violence on your fellow humans keeps popping up.

I think the paradox results from a confusion of static vs. dynamic equilibrium. The consequence of not fighting back when attacked is that you end up dead. But that doesn't mean everyone will fight back - people end up dead all the time, there are plenty of human behaviors that lead to the death of their host and still persist (suicide being the most obvious one). Empathy is pretty deeply rooted in the human psyche, and is adaptive under most peaceful conditions. It's not a stretch to imagine groups for whom it becomes a way of life, at least until they're dead.