The CC companies pass it down to the retailers in terms of transaction fees, who in turn pass it down to the consumers.
It doesn't matter whether your card gets stolen or not. It doesn't matter whether you even have a credit card or not. Everyone is paying for the cost of credit card fraud in one way or another. Even people who only pay with cash still bear the burden of credit card fraud, since they're still paying the credit-card-fee-inclusive price (unless they only shop at retailers with cash discounts).
Fraudulent charges are withdrawn from the retailers. Even if it has already posted to the bank account, they will take it back.
The CC companies never absorb the cost for fraud. Their greatest trick is convincing everyone they do.
In the end, consumers and businesses are on the hook for it.
I think this really depends on who you are. I find it pretty entertaining every time and enjoy trying to figure out where it was, knowing full well I just bitch at the bank and have little to no issue replacing the card.
(Yes, yes, the CC companies are on the hook for it, and you're not liable for any money charged to a stolen card, etc., etc., but it doesn't make it pleasant. It's stressful.)