Yep, even if the credit card didn't offer any rewards or protection, I would still use them because I'm billed a nonzero number of days after the charge happens. And that's an interest free loan, where I can instead earn interest. If your credit card balance is $5000, and you only have to pay it an average of 30 days after the purchases are made, well... The 30 day T-bill rate is currently 4.38%, so putting that $5000 into the safest, most liquid investment available for 30 days will yield $18.25, which is much better than cash or a debit card where you are immediately charged.
You shouldn't be keeping your savings in the bank, you should be keeping them in investments. When I charge something to my card, it's an average of 45 days until it gets debited from my bank account. That's 45 days that money can be in investments and profitable.
It doesn't matter much over just one or two months (and will be swamped by the direction of the market anyways), but over a lifetime it adds up.
Don't ever pay now when you can pay later, if you can invest your money during that gap.