There's no population upper limit if the soul can travel throughout the timeline. I.e. when it's done being me it can then become a cave man next or vice versa.