NYC is an archipelago separated by large bodies of water, and it is hard to avoid going through Manhattan to go between the western suburbs, the northern suburbs/Bronx, and the eastern suburbs/Brooklyn/Queens. What few links exist are always horribly congested.
Geology is the reason there aren’t skyscrapers in the west village etc though.
Downtown is where all the ferry landings were, and where a lot of the bridges were built after.
Midtown is where the rail terminals stop, so suburbanites can quickly walk to their jobs instead of paying a second fare for subway/bus/taxi.
There is no correlation between bedrock depth and height in Manhattan, as explained in the article of the grandparent comment. In fact, most of Downtown is landfill.
I think your causality is backwards. Why has downtown and midtown always been the two main population centers, and not one continuous city? Because that’s where the exposed bedrock is, with some swampy midlands in between in the West Village. Geology.