€5 seems high, but it's fairly common in eg Australia for smaller merchants (corner shops etc) to charge a card processing fee if your transaction is below a minimum threshold, usually $10.

The driver was probably trying to cover the transaction fee, it's not like most taxi drivers are raking in the big bucks.