It certainly can hurt your score. One small but real component is the average age of account.

Also, ratio of credit used to credit available is also a negative signal, which would increase when you close an account.

Those are somewhere between minor and middling factors. It's stupid we have to play the stupid credit score game, but a small decrease in interest rates can save you thousands when it comes to your mortgage.