The Japanese were closer to collapse than we realized, but if the invasion had started, neither side could have prevented major losses, even if both side's leadership would have preferred to. And we were days away. The war had a momentum of its own.
It's true that a misinterpreted signal has not led to retaliation... so far. We've been quite lucky. If Vasili Arkhipov had been a little more cranky on October 27, he could easily have been convinced to agree to launching a nuclear torpedo at American forces. Similar with Stanislav Petrov in 1983, although it's an open question as to how the leadership would have reacted had he informed them of the supposed attack. In 1995, Russia came within two minutes of deciding to "retaliate" in response to a scientific rocket launched from Norway. A slightly different trajectory or radar reading could easily have made the difference there.
Given the past 71 years, the risk of accidental nuclear war appears to be fairly low. However, we have no real idea if "fairly low" means the risk is 1% per year or 0.000001% per year. Both are consistent with what we've seen, but the former means that nuclear war is extremely likely in the medium term, while the latter means we can probably ignore it.
The fact remains that even a misinterpreted signal has not lead to the trigger of retaliation.
I'd even go as far to say that nuclear weapons have never been used in an act of aggression in a time of war. The war was already over when the US blew up Hiroshima and Nagasaki, what the US did was just ass-hattery.
But if you find it helps your moral psychology to worry about it, by all means, go right ahead. I'm going to continue to feel more concerned about being involved in a motor vehicle accident, if you don't mind.