As for the Nazis, their primary target was European Jews who had been living in Europe for millenia as well as other European peoples they viewed as inferior to the Germans, such as the Slavic peoples.
There is no possible narrative explanation of the war that does not centre on the scramble for Africa. Would there have been a great power war between Germany and the status quo powers at some point absent the scramble for Africa? It is very difficult to say. Had it occurred, it would have been radically different in shape. This goes back to my initial point about the uncertainty factored into any big counter-factual like this.
As for the Nazis, my point was that the very ideology of racism was a direct product of European imperialism. That it was primarily directed at Jews and Slavs is besides the point. Hannah Arendt - one of the most prominent Jewish intellectuals of the twentieth century - famously made the case that the Nazi holocaust was an outgrowth of European imperialism in The Origins of Totalitarianism.
It's worth pointing out that the one of the main rationales of Nazi imperialism was to create a living space in Europe in which to off-load Germany's surplus population. This had long been one of the main justifications for European colonialism, and the expropriation of Germany's colonies at Versailles made it look to Europe again to solve the problem. What are arguably the first two (imperial!) campaigns of the Second World War - Japan's invasion of Manchuria, Italy's invasion of Ethiopia - were both justified as means of creating outlets for their soaring surplus populations. Both of them faced profoundly racist immigration restrictions in the main destinations of migration, namely the United States and Australia.
As for the Second World War, I would say two things. First, that Nazi ideology was, in many ways, a product of European imperialism: its ideal of a Pan-Germanic state in Central Europe, its virulently racist and extermanist designs, and even its use of concentration camps. Second, one of the main grievances of Germany was that it was stripped of its colonial territories at Versailles, and that its territories were transformed into mandates run by the League of Nations according to principles of free trade, at the same time that France and Britain continued to run their own colonies as protectionist blocs. It was British sympathy for this colonial imbalance that, in part, stood behind the policy of appeasement. As for Japan, it was a classically imperialist power, reacting to the expansion of the US across the Pacific.