I feel like we're going around in circles here. So I'll try to explain one last time.

Most of the content about nuclear war in any LLM's training set is almost surely about how horrifying it is and how we must never engage in it. Because that's what humans usually say about nuclear war. The plausible sounding answer about nuclear war, based on probability, really should be "don't do it". So why isn't it?

Easy answer --- it only focused on "winning". It never bothered considering the consequences.

Similar lack of judgment is manifested by LLMs every day. It's working with memory and probability --- not to be confused with "intelligence".