It’s a combination of factors, one of which is that a prop is just out there, unshielded like the fans inside a jet engine (which still make a ton of noise in the case I’d high bypass models). Then there’s cortex shedding, which is mostly occurring along edges and tips. Airplanes also tend to fly at much higher altitudes except on takeoff and landing, or their noise would be an issue too. Finally you’re right that the number and arrangement of blades matters, because each blade produces some turbulence, vortices, etc which then interact with the next blade in sequence. All told the issue is just s high degree of aerodynamic noise. You get that with a wing and any aerodynamic body, and really if you’ve ever seen model jets fly you’ve heard the racket they make.

The major issue is that until drones came around, you didn’t have to worry very often about large number of high performance aircraft operating throughout civilian airspace, at very low altitudes in crowded areas.