It's this really raw, basic emotional experience which shapes our "core" understanding of ourselves and the world we live in.
"Healthy" attachment in this early stage comes from the caregiver being able to regulate the emotions of the baby. If the baby doesn't feel connected to the caregiver - such as if when the baby cries they don't receiving loving reassurance of their safety - then they form a dysfunctional template for social attachments and poor emotional regulation ability.
If the random babysitter has consistent and parent-like loving connection with the baby then maybe you're right, but I doubt that's the case.
Research apparently indicates that being loved in your first year is key to your development.
It could also be biting, jumping off of tables, hiding poo, poking holes in drywall, poking both ends of the dog, hitting sleeping parents with tools, and learning to scribble in books.
There's just a huge difference between a 3-month-old baby, who can't recognize faces yet and spends most of the day sleeping and eating, and a 3-year-old child who is running, jumping, telling stories, making friends, and learning to read. For a 3-month-old baby, I doubt that parents do much more than a random babysitter. For a 3-year-old child, there is a clear difference.