Yes, though your neighbours will probably also object to you putting up one of those big wind turbines in your backyard. Or a coal fired power plant.
Clearly a coal-fired plant would be an issue, to the degree to which they pollute the neighbours. There are diesel generators here attached to factories etc, and nobody really cares.
Objection to nuclear though is more visceral. With diesel I can hear the noise and decide if I don't like it. With coal I might notice the smell. But I don't think either would silently kill me. Nuclear leaks are invisible, and accumulative, so it can be (disproportionately) scary. Certainly those who are opposed to it will use fear to rally other people to their side. Fear is a strong motivating emotion that is easily manipulated.
You'd be surprised. There was a protest in my area against putting up five new wind turbines at an industrial site as the existing buildings are take down. People object that it would ruin their view... The current buildings is part of a massive coal fired power plant which is slowly being decommissioned and there are already three existing turbines.
You are correct that the obvious solution is standard production of modular designs are the solution to that.
This has all be true and obvious for several decades. Yet the realisation of this approach seems to still be far away.
It also doesn't factor in the non-technical hurdles, costs, and delays that are involved here. Environmental impacts, and local-population-resistance are significant, often terminal, factors in plant construction.
Put another way, nobody cares if I have solar panels on my roof. But if I have a tiny modular reactor in the garden the neighbours will (not surprisingly) object.
I say this as a nuclear supporter.