I can understand your concerns if folks are flying their RC helicopters around airports, but I'm pretty sure there are already airspace restrictions around airports for the same reasons you've mentioned. Beyond that, unless you're routinely buzzing rooftops, a low-altitude "drone" probably shouldn't be of any concern to your own flying, since you're going to be on an entirely different altitude vector.

I'm no pilot, but several of my coworkers are. Based on various conversations with them about flying (altitude clearances, ATC, etc.), I'd reckon you're severely over-reacting by calling for an outright ban on unlicensed RC aircraft flight. Altitude and airspace restrictions are reasonable; outright bans are not.