The point of contactless is to make it less bother to use cards. One of the tradeoffs for that is reducing the security – it's sort of a necessary condition. Losses are limited to maybe £150 in the UK (that's 5 transactions at the maximum of £30) without further authentication, and it's virtually certain that the card provider will refund these transactions too if your card is used without authorisation.
Contactless as an idea is probably ill-conceived though.
With chip and pin the customer was liable for all transactions when their pin number was used. It was your responsibility to ensure it wasn't exposed.
Now with contactless there is no risk to the customer, the bank/retailer takes on the full risk of fraud for contactless payments. You will not be out of pocket for a fraudulent contactless transaction. This is why contactless transaction are limited to £30 (initially was £20), it limits the bank/retailers potential exposure to contactless fraud, its also handily set to just about the average card transaction in the UK.
Contactless payments don't need a signature or a PIN, so they're significantly quicker.
1. I don't have to carry cash 2. I don't have to go to the bank to get cash 3. I don't have to count the cash when paying 4. No change to worry about. 5. Transaction logs via bank statement
The downside that this (OP) problem illustrates does need sorting and I was very sceptical of contactless before I used it, but its quite nice.
However as many have pointed out, there are potentially a boat load of skimming issues.