The issue is that Manhattan is not just a destination, it's also a transit - people often just want to go through Manhattan, and not into it.

If you removed all the through traffic, leaving just people who want to stop there, that change alone, would improve things dramatically.