Sustaining the city required a continual influx of residents. To be fair, a lot of the deaths were from slaves, but it was something of a deathtrap between the high heat, humidity, hot nights, disease carrying insects, flooding, and hurricanes. Were it not such a valuable port the city would probably have died out.