You don't get anywhere near the amount of control authority, and its speed if it depends on rotating the whole body of the aircraft weighting many tons.
Crosswind is really less of an issue near the ground, so you could first lower your altitude and precisely adjust later.
Crosswind is a bigger issue near the ground.
Yes, winds are slower at ground level, but the ground, and a crash into it is also closer!
That said, I was mainly thinking of wind speed as you said (IIRC ~ haven't flown in a long time, and only ultralight: as you approach ground, you need to compensate for crosswind, but near the ground you need to stop compensating), but was also picturing the possibility of creating structures to break wind. Isn't it done sometimes for helicopters? Hangars surrounding helipads?
No, helicopters can land in rather windy weather. They have lateral control authority, without having to change attitude, unlike quadcopters.
A bit of wind actually makes it safer for helicopters to land because it blows away the vortex ring https://www.youtube.com/watch?v=UBGZH8PzfTI
Try to land such thing in a crosswind without lateral control authority.
That's hard even with just 10kg quadcopters, and a "flying car" will be completely unlandable without extra rotor mechanisation, negating electric power advantages.