Do we have enough rare-Earth minerals to build this many solar panels? What do we do with the panels when they reach the end of their lifecycle twenty years from now? What do we do when it's nighttime in the Sahara?
To cover nights in the Sahara we use some of the absurd excess energy we get from covering the whole desert to make some Hydrogen or Methane and convert that back to electricity. If we don't want PV and gas synthesis for some reason to we can also just store heat in molten salt in solar-thermal installations and drive steam turbines. That's just a bit more expensive than PV today.
I'm not sure why rare Earth minerals would be a bottleneck. They aren't rare in the element abundance sense, just uneconomical to extract, which is why they are being extracted in countries where human rights violations are happening all the time.
The are no rare-Earth minerals in silicon solar cells.
The important shift btw is not from coal and nuclear to renewables, it's from fossil fuels for heating and transportation and industry to electricity. If we figure out how to build fusion power plants safely for cheap in a few decades, we will already have set up our infrastructure to run on electricity instead on carbon. Cutting energy usage is just a nice side effect of switching from 30% efficient ICEs to 80% efficient electric cars and from heating with gas to heating with a heat pump.