Can't speak for Atlanta, but the situation in NYC these days is pretty solid. Great startup community, lots of tech meetups, organizations like HackNY, places like General Assembly, etc. The list of cool NY startups is long, and includes Foursquare, bit.ly, Etsy, Hunch, Aviary, OkCupid, Tumblr, SecondMarket, Hunch, and more.

For Atlanta: Georgia Tech creates enough qualified computer students to feed a nice ecosystem. Some people do work for equity here, and there is even some locally available funding, (although nothing like NoCal). Housing is less than 1/2 the cost of SF even if you're living in the expensive parts.

One random other good factor: Atlanta has a huge imbalance of women vs men (as in, there are considerably more women than men). This skews even further when you take gay men out of the counts (which there are a lot of in Atlanta). This means its considerably easier to find women in Tech here than many places out west. (The same goes for several other minorities).

The imbalance is also useful for dating purposes too if you're interested in that part of things.

If your startup is related to music, this is probably one of the better towns for it (lots of hiphop labels are centered here). CNN is here, and Turner is a veritable fountain of contractor based work (if you need a bit of consulting work to tide you over). Also: Probably one of the easiest cities to host a conference in.

Don't forget Hunch :)

Heh, whoops. Well, Hunch is double cool. (It's on my shortlist of places to think about applying if/when the time comes.)

There's Coraid out in Athens, GA, a very successful startup in a very nice town.