The real world determines which geometry applies. That's the crucial difference. Sometimes physicists use mathematical reasoning to figure things out that map onto the world and later observation validates their reasoning. But observation can also invalidate, and then physicists have to go back to the drawing board or devise new experiments.