You do need to duplicate a lot of systems, but with modern electronics, these are a lot lighter and take much less power than before. The swarm aspect also allows for large synthetic apertures instead of a big single radar aperture in the nose of one aircraft. Swarms can also deploy cheap unguided weapons, because they can get very close to a target without the worry of losing a pilot and a $100 million airplane. The structural advantages of being small can also be exploited in an unmanned vehicle in that they can sustain much higher G loads than a large airplane, and with no pilot to black out, they will be much more agile in evading missiles.
The issue is not that any given drone is expensive. The issue is that any given manned vehicle with roughly the same profile will involve the same-ish hardware along with expensive life support and safety systems, and a super expensive pilot, and super expensive logistics and possible recovery missions. Plane for plane the drones let us do more with less, justifying their investment during ongoing appropriations. Human pilots will always have a role, but will never be disposable in the way a drone can be.
And, yeah, "swarms" because there is no cookie cutter monolithic answer to what unmanned military craft should look like... Low speed ground hugging smart missiles, ocean based spy capsules in mesh networks, slow speed support craft, high altitude recon craft, in theatre air fire support, air-to-air combat drones, and orbital UAVs can all coordinate. All we know is ever increasing automation and standardization is intersecting with decreasing costs and requirements... very fertile ground for mass production.
Why would we shrink drones down? You read drone swarm and assumed small, but most swarm proposals are using drones of a similar size as today or even large in some cases. They're swarms because of the way they interact with one another, and overwhelm enemy defensive systems, not because they're small.