The fallacy that, IMO, we are falling into is (ironically) caused by scientific reductionism... the simplest explanation is probably the correct one. This doesn't work as well for the evidence types paleoanthropology has to work with.
Take Israel or Morocco as a comparative example. There were sapiens there before Neanderthals, but they are not thought to be ancestral to any modern people. The majority are not. This makes intuitive sense when thinking of species or subspecies (eg Neanderthals), but it works the same with populations.
Also, it's easy to think of ourselves in permanent "infinite growth" mide, where we colonise and dominate. But, recent years aside, humans bloomed and receded just like other species. Range expanding and contracting.
Just because a population is in the same place does not mean that it's ancestral. The first, second or eighteenth population may have no current ancestors... or none in that area.
As Dawkins puts it "descendants are common, ancestors are rare."