The infringement is limited to a very specific class of people who are a small subset of the general population. Consider that convicted felons may not own firearms, is that an infringement? I would argue that it is not, as you lose certain rights when you are convicted, just as you lose certain rights when you are deemed to be mentally ill or of unsound mind.

In other words, we agree "shall not be infringed" has exceptions, and are just haggling over where the line is.