You also had millions of years for diseases to evolve to infect people in the Old World. Then there were fairly small populations that traveled to the New World. If they didn't bring the diseases with them, there was only about 10,000 years for disease evolution, and a much smaller population for much of that time.