It's just different. From whole to 2% to skim you have big differences in how rich the milk tastes. Once you get used to one, it's hard to go back to the others.

I always drank whole milk, so skim tasted like water. Then I started only drinking skim milk, and whole tasted like ice cream.

I don't have any dairy issues but it's very, very rare that I drink milk anymore these days, but I still enjoy it.

It tastes completely different, much thicker, much rich. 2% tastes almost like water compared to whole milk.

The first time I had whole milk was at an after school program I went to in elementary school. They fed us a snack first thing when we got there. One day they gave us (whole) milk with our snack. Almost everyone (including me) took a sip and complained the milk was bad/tasted funky. The head person of the place came out and told us the milk was fine, we were all just not used to drinking whole milk.

I still don't like the taste of any cow's milk too much, I prefer unsweetened soy milk. Soy milk also lasts forever in the fridge which is a big plus for me.