Minimum records being broken can mean more variance causing outliers more frequently, wouldn't necessarily mean that they push the average down as they might be short events followed by a different weather pattern.

The disruption of the polar vortex drops temperature significantly across Northern Europe but it can be caused by the warming of the atmosphere so even though it can cause record lows it is a short lasted event and on average the temperature is still going up.

That is also a side-effect of climate change, more powerful bursts of extreme weather.