I can also argue increased demand for solar panels increases manufacturing and lowers prices over time making solar more competitive. It's not clear that it's a net negative.
This has been the effect we saw when Germany started investing so heavily into solar.
Manufacturing them is not environmentally neutral either.