Fertilizer run off from organic farms contains just as many nitrates as metal^h^h^h^h^h^h non-organic farms. Those are the pollutants that cause the biggest problems in water ways and generally encourage algal blooms that kill other plants and animals.

Switching to plant based agriculture would require even more fertilizer to be used as the land area needed limits super effective crop rotation.

Other problems are plants are much less tolerant of different environments than live stock, so you have to more aggressively farm the limited area. Many of the ideal areas for growing plants also lack nutrients or even water (CA is essentially a desert but grows most of the worlds cereals because the weather, soil are excellent for growing, even if you have to pump the water in from elsewhere...)

Of course this ignores harmful effects of livestock, and is just meant to illustrate “not livestock” does not mean “good”.