"to hold a country" is the key phrase there
Exactly. Why do you think China insists on celebrating diversity (the 55 official ethnic minorities get lots of airtime in cultural events - I've seen a national day celebration on TV in which every official minority had a representative dance troup), while emphasizing some kind of fundamental unity of the country (at the end of the celebration, all the dance troupes danced together).
As for geography, China is very flat. It was flat 5000 years ago, and peasants (the occasional ruler with a civil engineering fetish) have been making it flatter for millennia. There's nowhere for an army to hide. Dissenters can hide in the concrete jungle, but you can't make any real strongholds. Also anyone who holds the Yellow River and Yangtze River will have strategic control of most of China, and the two rivers have been linked since (IIRC) the Qin dynasty.
As for the "small towns" being a little, well, dead, that's partly a result of rural poverty (and taxes redistributing wealth to the capitals); but also (as you said) of the bigger cities being of interest to young ambitious people.