Half of the cards I remember activating were based on the originating phone number (which I think you could still possibly spoof) which should have been tied to your account when you signed up. If the phone you're calling from isn't the same one on your account, there's likely some additional information you provide or maybe go through a CS rep to finish activation. Unfortunately, like you say, there are also many bank's systems who ignore or simply don't care about the call ID number.

As for interception, I guess you could require a signature for delivery but that's a pita. You just have to weigh the risks in that situation and the convenience of not requiring a signature obviously outweighs the risks of theft.

EDIT: grammar