Those panels could still be deployed elsewhere and produce power for the grid.

Manufacturing them is not environmentally neutral either.

I can also argue increased demand for solar panels increases manufacturing and lowers prices over time making solar more competitive. It's not clear that it's a net negative.

This has been the effect we saw when Germany started investing so heavily into solar.