Deaths per joule doesn't account for the safety issues if nuclear power were more widespread. The deaths caused by weapon proliferation and the long-term ecological impact of more nuclear accidents doesn't scale linearly. We can handle an accident every decade or so. But if nuclear power replaces fossil fuels, we're looking at closer to one every year. It's unknown what the long-term consequences of that are.