Immune systems don't weaken from a lack of exposure. It's not a muscle that needs to be worked out. Besides which, your immune system is pretty much constantly at work with relatively benign foreign tissue regardless of whether you're wearing a mask or not.

Some things like the uptick in RSV can be explained by children being initially exposed to RSV a bit later than usual, but for your average adult that's not really relevant.