I think some reference is useful, here.

Agriculture is and will be much more land-intensive than solar.

I once calculated that the US could switch entirely to solar just on a fraction of the land that is currently used for ethanol production alone.

And what is a better place to put solar panels? In the desert or fertile arable land? Deserts contain life, of course, but the density of life and biodiversity per unit area is much smaller. Additionally, solar panels provide shade and, unlike agriculture, don't require intensive tilling, harvesting, irrigation, spreading of pesticides (natural or otherwise) or fertilizer and the runoff that causes. They mostly just sit there, maybe getting a cleaning every once in a while (but unlike concentrating/thermal solar, don't need to be cleaned super often).

Once the panels are placed, your desert tortoise can nestle in the convenient shade they provide.

We'll destroy habitats anyway. But stopping/slowing down climate change by reducing the habitat of some species will still be beneficial for global diversity of species. I find it hard to believe that warming oceans would kill less species than solar panels in the desert. And even for 100% renewable energies, only small parts would have to be used for production.