Anyway, as a simple test try and count how many cars you can see next time you go for a drive, how many people’s gender you can infer, and how many you can positively identify based on their looks. It’s significantly safer to try this as a passenger.
Might be interesting to try, but I think in a situation where I notice a bad driver I'm much more likely to identify their gender than in a random passing car.
But I live in a comparatively small Austrian city riddled with traffic lights, weird intersections, crosswalks, and way too much traffic in general, so I get ample opportunity to look at slow moving cars. If you're American your culture around cars is probably very different and that might explain why we seem to have a different intuition about this.
Now sure, the smaller percentage may still add up to a significant number. But people do a crap job of limiting their mental tallies to only the people they actually verify. If they assume someone is say drunk their going to add that as evidence even if they never actually have any direct evidence for drunkenness/age/gender/ethnicity etc.