Sunlight does not bring in a lot of IR, and almost all of that is absorbed in the upper layers of the atmosphere. What you have to consider then is ambient IR radiation from the surroundings due to thermal emission. The process is the same as the one we're assessing for the PVs, and the wavelengths are not too different. PVs are normally warmer than the surroundings due to their higher temperature, so less energy will be absorbed that radiated off in the IR.
Hm, maybe I misunderstood how you want to remove the IR? Let's consider some sort of reflectant shell around the PVs but not part of them. On the inside it would receive the heat from the PVs and of course, also reflect great parts of it back, letting it accumulate. We built an insulation. Not good. Maybe you can explain your idea in more detail?
The mechanism I discussed above is of course not the only relevant process, things like heat conduction and heat capacity of the material are also important.
In conclusion: Yes, there would be a net loss of incoming energy, but the mechanims that let energy be transported off might be severely hindered by our alteration. Thus, an accumulation of heat.
There might be ways to make it happen nevertheless, eg with water cooling, but it would require more infrastructure. (But then, I'm not an engineer..)