Because I don't see how the phone can be much quicker when plastic cards support the same contactless transactions, and have no buttons to fiddle with.
The phone can be quicker than a plastic card because the phone is more convenient to reach. It's just in my pocket. Reach in, thumb on home button, put near terminal, approved, put phone away. With a card, I have to get it out of my wallet too. Not a big deal, but not as smooth. (Plus, with today's addictive smartphones and short attention spans, half the time my phone is already in my hand at this time.)
Strictly speaking you can leave the card in your wallet and just hold the whole wallet to the terminal. But this only works if you have exactly one contactless card (or if you don't care which card pays). Unfortunately the US is moving away from such cards. I used to have two or three, but I just checked my wallet and out of the eight payment cards I have in there, none of them support NFC. Sad.
Unfortunately, Apple Pay is suffering from the wrong end of network effects. Support is far from ubiquitous, so you don't know whether any given transaction will support it. You can look for the NFC logo, but some places with the logo still don't support it. At that point, it's easier to whip out the card by default. If I've been to a place before and know they take it, I'll usually use it, but there aren't many such places for me.