It's just different. From whole to 2% to skim you have big differences in how rich the milk tastes. Once you get used to one, it's hard to go back to the others.

I always drank whole milk, so skim tasted like water. Then I started only drinking skim milk, and whole tasted like ice cream.

I don't have any dairy issues but it's very, very rare that I drink milk anymore these days, but I still enjoy it.