Well yes, it saves people time (and time is money), but it's a trade off and sometimes ends up costing more than it saves.

If your kid wants to buy something, normally you just give them cash and they can't spend anything more. If you give them your card, now they can spend as much as they want ($1000 on Clash of Clans for example). While there are solutions such as giving your kid their own card you transfer the money to, it's a lot more effort than just pulling out a $5, and I'm sure many parents would forgo that route and just hand the kid their card.

What do you do when you want to buy something anonymously? Normally, you'd just pay with cash. Want to donate to Wikileaks without anyone knowing? Use a money order. In a cashless society, you can't do it as simple as that. You need to purchase prepaid debit cards (which costs time). And if the prepaids can be linked with the purchaser, then you'd need to spend even more time buying bitcoins, tossing them in a tumbler, then buying prepaid debit cards. A lot of extra work compared to just using cash or money order.

So yes, a cashless saves time, but it also costs time in other areas. Has Sweden found a balance? Doubt it.