Crosswind is a bigger issue near the ground.
Yes, winds are slower at ground level, but the ground, and a crash into it is also closer!
That said, I was mainly thinking of wind speed as you said (IIRC ~ haven't flown in a long time, and only ultralight: as you approach ground, you need to compensate for crosswind, but near the ground you need to stop compensating), but was also picturing the possibility of creating structures to break wind. Isn't it done sometimes for helicopters? Hangars surrounding helipads?
No, helicopters can land in rather windy weather. They have lateral control authority, without having to change attitude, unlike quadcopters.
A bit of wind actually makes it safer for helicopters to land because it blows away the vortex ring https://www.youtube.com/watch?v=UBGZH8PzfTI
Crosswind is really less of an issue near the ground, so you could first lower your altitude and precisely adjust later.