If it were simply a matter of losing particles, I'd expect that one of the many very capable metrology labs around the world would've devised an appropriate experiment, e.g. storing two exact same masses at slightly different temperatures near absolute zero and characterizing the difference in mass (since presumably the higher temperature sample would lose slightly more mass if it were down to high energy particles).

A test for this theory would have to involve a means of accounting for these kinds of fluctuations.