Yes, but remember the "bottleneck" from 70,000 years ago, when the human race was nearly wiped out. At that time, because of a nuclear winter brought on by a volcanic supereruption, there may have been fewer than 10,000 humans in the entire world.

http://en.wikipedia.org/wiki/Toba_catastrophe_theory

So, based on that, we're descended from fewer people in the past than most people realize. Another way to say it is that most of our ancestors were cousins.