Remaining small but alive for thousands of years is not entirely unthinkable, but it's very unlikely.
Also typically a population grows depending on birth-death rates which reach quick limits in places of low resources. This is self-evident from different population sizes in different geographic habitats. Instead human populations mushroomed only in select places. See Guns, Germs and Steel by Diamond for more.
Even if your colony is surrounded by impenetrable deserts, I'd think you could always propagate along the coast, living off the ocean.
That I'm just some internet guy speculating without any deep knowledge is probably obvious, but let me make it extra clear!
Living off the ocean is also incredibly dangerous.
Well, logistically limited by the carrying capacity of the local area with available technology, etc.
But logistic growth looks exponential...until it doesn't.
That is, if there don't arise artificial limiting factors, like a cult/religion rule or system that takes control over population's dynamics. When that happens, it may enter in a stable state until an occurring external disruptive change.