Under the exponential growth model, the increased incubation time (before you hit the same total population size) is inversely proportional to the log of the initial dose, so (e.g.) 1/100th of the initial dose gives you only ~7 (log2(100) = 6.64) additional doubling times before arriving at the same population size. While this is not insignificant, it does mean that initial dose is potentially much less important than sources of variability that affect the in-host doubling time.

Edited to add: Here's a preprint that is relevant to the "initial dose vs. immune system response" thought: <https://www.medrxiv.org/content/10.1101/2020.03.26.20044487v...

The math is a simplification of a biological process that is highly variable and poorly documented. Your original idea is a fine hypothesis, though the suitability of the params behind exponential growth is mostly irrelevant.

We’re learning a lot about virology during this time. Infection and immunity are not binary, and now we have enough data to recognize that. We will also learn a lot about mutation, things like carrier recombination. I think this will change everything about how we attempt to control things through vaccination.