But New York is huge and you can find a little bit of everything. Brooklyn alone is larger than most large cities, with 2.5 million people (over half the size of metro SF, and about the same as metro Pittsburgh). Everything gets much, much cheaper when you leave the ultra-density of Manhattan. It's like any other city center, just scaled way up.
Also, DUMBO in Brooklyn is fairly easily accessible and is becoming quite a startup hub. NYC seems expensive, but if you look hard enough you can find good deals anywhere.
Don't be afraid to look outside of the manhattan/williamsburg/dumbo/long island city areas, either.
Most people who think that NYC is incredibly expensive (I've even gotten this from folks who live in La Crescenta, CA) haven't bothered to explore beyond Manhattan.
Of course, I disagree wholeheartedly with Thomas on Chicago unless you're bootstrapping. If you don't want to do SV, Boston and LA (especially now with Jason's Open Angel Forum events) are pretty good options.