Because in the winter the thermal energy within your room will “flow” out of those “cold spots” at surprisingly high rates; in summer those spots act as space heaters. In both scenarios they will adversely affect the ambient temp of the room. In short, whatever method you use to heat/cool the room will have to work harder and will take longer. Here’s [0] a good overview of heat flow in this context, with some hints towards the math behind if you wish to go deeper.

[edit: add link]

[0] https://www.energyvanguard.com/blog/Flat-or-Lumpy-How-Would-...