One cup of milk is about 242g, that means milk has energy density 0.62 calories per gram, much less that sugar (4 calories per gram) or fat (9 calories per gram).

Also, it looks like 80 calories will not change fasting state of your body (see above about 20g of BCAA). You may put 80 calories / 0.62 calories per gram = 129 grams of milk or half of cup, into your coffee or tea and not interrupt your fast.