You will care when someone steals your number, although it likely won't be from the card.

Banks will often make you jump through hoops when something goes really wrong, even when they are at fault.