I love Savannah and Augusta. Those cities are a lot of fun. I spent time living in both in my early 20s. The segregation both is and isn't there. It's very strange. I think that the feeling you're describing is due to the ghost of slavery. It was always very hard for me to separate the history of the antebellum south from those two cities specifically.

To be fair, though, the most segregated place I've ever lived was easily East Los Angeles. It doesn't have that same feeling that you describe, of course, which makes it ... different but in a slightly better way.