There's nothing stopping a group of motivated people from building a drone assembling factory that fits in a cargo container.
There may come a time where we have to stop outdoor festivals and parades because of people like that Las Vegas shooter from a couple years ago.
The real concern is when people can build a factory that can build the factory...
The only problem is that the novelty of the approach would be such, that it would bring too much visibility to the attack. Most bad actors don't want to be detected: "he fell from the window" or "his plane crashed" leaves the door open to coincidental causes. Drones are noticeable, particularly in swarms. They would be useful only in "showy" attacks, i.e. open terrorism, a tactic that is largely out of favor after 20+ years of substantial failure.