>>>> Possible toxicity of something that we deem safe could be an explanation.

Probably a minor contributor. As the paper states:

According to Lokeshwar, potential causes for these declines could be increased obesity/BMI, assay variations, diet/phytoestrogens, declined exercise and physical activity, fat percentage, marijuana use, and environmental toxins.

More like lifestyle changes.