But I'm thinking if it hits a pole it could release CO2 and permanently affect the temperature.
It's still unlikely. The universe tends to be stable, because if an impact could heat up Mars, it would have, over cosmological time. This is not Mars' first impact.
Our Earth intuitions are really wrong for dealing with the rest of the universe. The rest of the (known) universe doesn't have intelligent critters on it making active and wild changes in very short periods of cosmological time. The universe is generally full of stable systems, in the physics sense of the term "stable". To the extent that you think otherwise, it's because we humans are interested in the unstable bits and they get a lot of play in the press, but we know and talk about them because they are the exceptions.
Transients on this scale, however, could last hundreds of years. Even on Mars, CO2 takes a while to escape from the atmosphere; on human time scales we'd see a new metastable Martian temperature.
(Assuming a bunch of CO2 actually were dumped into the atmosphere, that is.)
When we talk about impacts changing climate on Earth, it has nothing to do with the instantaneous heat increase itself. It has to do with how the impact can change the rate at which Earth receives and retains heat, by changing the atmosphere. And Mars hasn't got very much atmosphere to change, so even that isn't much of a factor.