> Somewhere in between there must be a gradual cutoff where it becomes “too much”.

There's a huge range of dimensions beyond how many people: Who is allowed to immigrate? How long do they get to stay? Do their children become citizens? etc.