In other words, cash buyers are subsidizing the interchange fees, your Rewards Points, Cash Back deals, etc.
This is evident especially at gas stations. Many have "cash only" prices that are lower than credit prices; Arco generally has the lowest gas prices but accepts only cash or ATM (with an additional ATM fee).
It's been policy for a while now that you simply can't charge more for (just) credit card transactions (you could however discount cash purchases). That landscape is changing recently [1][2], but we haven't seen its full effects yet.
[1] http://www.dailyfinance.com/2012/07/19/3-reasons-why-credit-... [2] http://www.dailyfinance.com/2013/01/24/new-credit-card-check...
I'm not sure how/if this was recently changed in the US, my knowledge is 5+ years out of date now and I'm not in the US either.
The logistics of drop safes and daily deposits plus losses due to counterfeiting, robberies and pilfering can cost a similar amount to the 3-4% credit card fees.
That's why merchants aren't grumbling too much.