If you look at the statement you yourself make, it requires a lot of assumptions:
* Interstellar colonization is possible (this is explicitly taken as a hypothesis, though)
* The ability to colonize another star system is the ability to colonize a large number of star systems. This is to say, the social consequences of colonization are not going to grow larger with size. A possible counterfactual would be that large civilizations might put more energy into securing control of known systems rather than embarking on further colonization--and the anecdotal evidence from the history of Earth is that this is indeed the case.
* Colonization is a logistic process (which is probably what you really meant when you said exponential, but the difference is a technicality for this discussion). If interstellar travel is really as difficult as our knowledge of physics suggests it to be, then interstellar colonies are likely to develop as completely independent civilizations without maintaining strong ties to their parent motherlands, and subsequent colonization efforts are going to include attempts to "colonize" already-colonized systems and consequently logistic growth wouldn't be achieved.
If you can do any colonization at all it shouldn't be that much harder to colonize the whole galaxy. Once you've got the ability to colonize exponential growth will take over and pretty soon* you'll be everywhere.
*In the context of the Fermi paradox, we're wondering why no aliens have already colonized the galaxy (including our solar system). So by "pretty soon" I mean relative to the time interval between when such aliens might have appeared and now.