The brain size needs a huge head, so in a sense it is.
All the factors are interwined. The growth of the brain seems to have been a very fast evolution, The pelvis had no time to adapt, so giving birth early was the final solution.
With that in mind, I don't find fair to shame the pelvis, or the brain. The weakest link is the legs, but it's not their fault either since they had no time to mature.
Give one of those cocky foals a head of proportional weight and allow them to use only two limbs and see who's stumbling now!
Edit: newborns can't even balance their heads on their necks, they're extremely fragile.
The balance comes much later than the strength.
If you start occasionally holding your baby up by the thighs when they are 4–5 months old (their back/abdomen muscles at that point are typically plenty strong), after about a month they will have the balance to hold their torso upright when sitting on an adult’s shoulders, for at least a while. By age 8–9 months they will be very stable and good at balancing for long stretches of time. (I recommend any new parent try this, not only for the baby’s sake but because carrying a baby on shoulders is dramatically less tiring for an adult than carrying a baby in arms, and less hassle than a stroller or baby carrier.)
People generally hold their babies from the top (e.g. with hands in armpits), in a way that they are passively stable. If you want to train someone’s balance, you want to force them to actively stabilize themselves from a default-unstable position at least a bit every day. Babies have very short torsos and small heads compared to older kids, with the effect that their torsos have a much smaller moment of inertia about the waist: it takes several times more muscle strength for a 3-year-old to lift their torso up.
Do you mean baby sits on my shoulder with his legs around my neck, I am holding on to his thighs, and he can hold on to my head with arms? We both face forward in same direction.
Then after a month or two of practice (a few minutes per day, nothing too serious here) you put your baby on your shoulders, holding his ankles, and he will be able to hold his torso up with his abdominal muscles for at least a few minutes at a time. Practice this for a few more weeks and you will be able to walk around town with the baby on your shoulders.
If you get a few 3-year-olds and e.g. get one to practice kicking a ball for 10 minutes per day, another to practice riding a 2-wheeled scooter, another to practice hanging from monkey bars, and another to practice swimming, after a few months there will be a wide gulf between their abilities at those skills.
Anyway, hard to conclude, just think your logic is missing a little.
Have you ever seen a new-born baby's legs?
They aren't even remotely strong enough to walk, no matter what the brain tells them. It takes those months to develop leg muscles. New-borns don't have them.
It's absolutely not restricted by the brain size.