These papers are good references:
Furthermore, "Milk", quite surprisingly, is not well defined either. I've switched to RonnyBrook recently (NY local, organic, grass fed, and I get it within two days of leaving the cow), and it's a world of difference: The taste, my body's reaction (My lungs and sinuses would often respond to milk by clogging and excreting stuff, they don't anymore).
I haven't been able to find high quality sources (peer reviewed, etc) but I've read claims that the nutrient and vitamin profile of milk from grass fed vs. corn fed cows is completely different - which I don't really doubt - although I'm waiting for some high quality analysis for the exact difference.
Cutting out vitamin d rich sources and most calcium rich sources had me back to normal within a few weeks. I'm back on calcium rich sources now without, much problem, but still need to watch for high vitamin d sources or the issues return.
It's a real pain in the tail as the number of food products fortified with calcium and vitamin D is remarkably high.