I am not an expert at the engineering or physics, but I did work at a leading PV manufacturer for 8 years, and I was amazed at the effort required to coax even the tiniest improvements in conversion efficiency out of the devices in a way that is both manufacturable and a benefit to the cost-per-watt. Then after all that, every PV platform has a theoretical limit anyway. So the whole enterprise seemed like pulling teeth. Especially when you think about how demand will increase in the future.
On the other hand, it seems like with nuclear the potential energy is so overwhelmingly high that the primary effort is holding it back (hence the dangers) - not trying to squeeze out tiny drops. Given this, just intuitively, nuclear seems like the preferable starting point. We can deal with the obstacles as we go. Seems like humans have done a pretty good job at turning what once appeared to be insurmountable risks into things we take for granted. Skyscrapers and air travel come to mind.