And it makes a big diference— as I mention below, our most recent common ancestor is probably only 2~5,000 years old. However, genetic testing puts the age of our most recent patrilineal ancestor ("Y-chromosomal Adam", the one man every man alive now can say was his father's father's father's father's...) at more than 60,000, possibly 140,000 years old, and our most recent matrilineal ancestor ("mitochondrial Eve", the one woman etc.) is ~200,000 years old.

The point being that after a while, everyone is related from a mixed lineage, but male-only or female-only lines may die off, either a man having only daughters or a woman having only sons, or having no children at all.