Genetic data is misleading; it only means that number of people had surviving descendants. It's entirely possible that the population never dropped below N, but one particular tribe gradually became dominant and genocided everyone else. Remember that endogamy is the default!