It really depends on the kind of factory. I live in a residential neighborhood with 'factories' right down the street. The factories produce small batch food, artisanal crafts, etc (there's a lovely chocolate factory 10 blocks away). Many sell to large audiences, or have national brands. I don't see what the big deal is at all. The neighborhood is lovely. It is more walkable than 95% of American neighborhoods. It's a historic neighborhood with lovely trees, beautiful architecture, and lovely ornamental details. It is highly priced and inhabited by well-to-do people. There is a natural limit on factory growth (namely lack of space). If a company outgrows its space, it'll have to move somewhere more appropriate and let a smaller growing business occupy its space.

Why fear such factories?

Of course, factories with toxic waste and pollution? Yes those should be built far away from people, but actually, my parents, who live in the suburbs are closer to that sort of thing than I am even though there spot feels more 'open' due to zoning.