Yes, but in this case I'm referring to formula, food, or breast milk that's been sitting in a fridge.

You don't need to "heat" it, but some babies are more receptive if its not cold.