The last time I looked into this what I got was that there'd been a statistical error made in the process of determining the RDA, and it was just never corrected. The RDA is the amount that guarantees that only a certain fraction of the population has a deficiency, if everybody eats that much, but they calculated it so that only a certain fraction of population studies would find a deficiency on average. Furthermore, dietary vitamin D is absorbed rather poorly, so the amount of it you have to eat is immense compared to going out into the sun, resulting in really high "recommended" RDAs---it's the blood concentration divided by sensitivity but the measured sensitivity is surprisingly low. [1]

[1] A Statistical Error in the Estimation of the Recommended Dietary Allowance for Vitamin D, Paul J. Veugelers* and John Paul Ekwaru https://dx.doi.org/10.3390%2Fnu6104472

Also, isn't milk consumption unevenly distributed across the whole population? E.g. those with an Asian background I would expect to consume less milk than the population average.

True, I think food supplementation is hard to do in general since the dose needs to be appropriate for anyone that may consume it, with age ranging from infant to elderly.