You're forgetting that West Chester is the county seat of Chester County PA, the fastest growing and richest county in the entire state (and usually somewhere in the top 25-40 of the US.) So, you have the county courthouse and all the typical government stuff, which Downingtown lacks.

Lancaster is much larger than either, though different cities have varying land areas they count. Lancaster is also the seat of its same named county.

I look at 'college town' as defining somewhere where school(s) are located, and the population count of 18-24 makes up almost 50% of the population. Newark, DE (University of Delaware) and Bloomsburg, PA (Bloomsburg U) seem to fit that better.