The template for this ofc is how we handle (or don't handle) the danger posed by people inside cars, to people outside cars. In aviation we will do a lot to avoid even one death, in the air or on the ground. But in cars we mostly don't give a crap. It took decades for drunk driving to become unacceptable, but outside of that, we are still pretty ok with death by car. The only survivor of the collision just has to claim the person outside the vehicle "darted" and we all shrug and move on.
I would just love it if we could apply lessons of the past to new technology. Instead we just hand wave it all away. Then in a few decades, if enough people die and their surviving loved ones invest enough time & energy, maybe we'll make a few tweaks to the formula.
So no, we don't need 5nm separations for 20 lb drones. But we do need some sort of structure that recognizes the people under them didn't sign up to be part of the beta test. For bonus points it should also recognize that externalities exist and should be priced but I am not holding my breath.