Reducing fat means it takes more of it to get the feeling of being full. The milk is also full of growth hormones and carbohydrates which we get too much of, so drinking more of it is potentially worse than a bit more fat.