This is why very progressive countries implement combined paternal and maternal leave. If one spouse doesn't take it, the couple loses that time.

The pressure on men and women to take time off after a baby diminishes the effect you speak of.