The infringement is limited to a very specific class of people who are a small subset of the general population. Consider that convicted felons may not own firearms, is that an infringement? I would argue that it is not, as you lose certain rights when you are convicted, just as you lose certain rights when you are deemed to be mentally ill or of unsound mind.