Fair point in theory and I'm not familiar with the literature, but I'd guess at least some researchers have studied ways of controlling for this: eg, looking at cases where father dies early and mother does not remarry, single mothers who adopt or do artificial insemination, etc.