There's just a huge difference between a 3-month-old baby, who can't recognize faces yet and spends most of the day sleeping and eating, and a 3-year-old child who is running, jumping, telling stories, making friends, and learning to read. For a 3-month-old baby, I doubt that parents do much more than a random babysitter. For a 3-year-old child, there is a clear difference.
It's this really raw, basic emotional experience which shapes our "core" understanding of ourselves and the world we live in.
"Healthy" attachment in this early stage comes from the caregiver being able to regulate the emotions of the baby. If the baby doesn't feel connected to the caregiver - such as if when the baby cries they don't receiving loving reassurance of their safety - then they form a dysfunctional template for social attachments and poor emotional regulation ability.
If the random babysitter has consistent and parent-like loving connection with the baby then maybe you're right, but I doubt that's the case.
Research apparently indicates that being loved in your first year is key to your development.
It could also be biting, jumping off of tables, hiding poo, poking holes in drywall, poking both ends of the dog, hitting sleeping parents with tools, and learning to scribble in books.
And how much in the first five months?
My understanding of the research around personality development is that something like 80% (BS statistic I know) of the personality is formed in the first 5 years of development. The early few years are when we get conditioned with the core emotional programming: "I am safe" vs. "I am at risk" - "I am lovable" vs. "I am unlovable." etc. which plays a huge role down the line in lifestyle, relationships, and life in general.
Probably not a big deal whether it's the father or mother at the young age, but having adults around to be available to the emotional needs of the young child seems to be extremely important. Maybe easier to split the "full time job" of parenting in this critical period?
If you want to nerd out, check out "Affect Regulation and the Origin of the Self - The Neurobiology of Emotional Development" by Allan Schore