Agreed, it's actually just impossible to have a non-polarized discussion if people identify too strongly with one of the poles. This isn't unique to HN, we essentially live in a bipolar society. Even if you present a middle ground, someone identifying with either pole will assume you are identifying with the opposite pole, because they see that part of the world in black and white.
Nevertheless, these discussions are good for clarifying your understanding of the world, just try not to expect much in the way of respect, appreciation, agreement, or understanding.
This is a good time to bring up one of my favourite Paul Graham essays, "Keep Your Identity Small".
http://www.paulgraham.com/identity.html
Religion, politics, professional sexism, all these are topics that infringe on people's identities, causing a strong gut reaction that leads to terrible discussions. Even programming languages can end up in a similar category, if you have too many people that identify as a "C++ programmer" or a "Pythonista", or a "Lisper", which leads to the oft-seen programming language flame war.
That's a good point; an effective compromise may be to allow heavily flagged discussions to continue after being pushed off the front page, instead of [dead]'ing them (which halts conversation abruptly.) This is actually what happens currently most of the time (as far as I have seen anyway) and I don't think it is particularly bad. The people who are already involved in a discussion can continue it, but more people are not drawn into it as the poisonous discussion doesn't stay visible on the front page.
Unwilling? No, HN is more than willing to have a good debate. The issue is that with some particular topics, HN is unable.
HN is not a perfect platform for intelligent discussion. It is unable to handle certain types discussions and the only effective mechanism it has to keep those discussions from consuming the rest of the site is to crudely cull them.
Humbly, I suggest that if HN's inability to discuss some things surprises you, then your opinion of HN is too high. It is imperfect, very imperfect.