I'm not sure that line of argument holds up terribly well; people working in an industry I consider immoral aren't really my primary concern. How would you respond to an analogous argument like "landmines are made by people in factories - if we stopped making landmines, all those poor people would lose their jobs, is that what you want?" (side note, the answer to that hypothetical is also 'yes').