USB 2.0 spec only guarantees 100 mA to be available before active negotiation (which is most likely not enough to do any charging at all), and 500 mA afterwards (which is very slow). For more, you're always relying on non-standard signaling of what the charger is capable of. So, it highly depends on the used charger whether your phone charges fast or not and you can probably easily find 12V chargers that charge slow and USB chargers that charge fast.

No doubt there are some USB ports that charge quickly — my point is that many of them scarcely charge at all. This is pretty dumb, since the whole point of a USB port in a car is to charge a phone or tablet. I would have guessed that they would put in a port that is capable of charging at least as fast as the 12V adaptor, but apparently this is not always true.