And yes, socioeconomic factors play a huge part in outcomes.
If the schools were set up more like Montessori schools, this argument against homeschooling would carry a lot more weight. Age-based cohorts are pathologically bad environments, and if you look at history as a whole, a bizarre exception to the normal human environment. If it wasn't for the fact that the people arguing happened to grow up in that environment and thereby consider it normal, despite how abnormal it actually is, nobody would be arguing it's a good idea.
I was educated montesorri until middle school, and while there was plenty of socializing it was ultimately detrimental imo. A couple of us including me had serious trouble getting assignments done with all the distractions and were always in the principles office as a result. One kid just had to work in a separate 'office' because he could not actually get his stuff done with his friends there in the class room available to talk all day. I didn't get my work ethic up until high school, didn't really apply myself until college, and to this day I struggle with self discipline and staying focused on the task at hand.
So, you went from what I describe as a reasonable environment, where you did OK, to what I describe as a pathological environment where suddenly you had a lot of trouble, and apparently mostly (but not entirely) recovered when back out in a more reasonable environment.
This is basically what I would predict with my theory. You were damaged by entry into a pathological environment. How would you be if you had never been exposed to it? How much more damaged are those who never knew anything else?
I don't think your experienced is evidence against my point... it's evidence for.
You're not learning explicit skills. You're learning implicit, semi or involuntary communicative behaviors. Similar to those not learnt by animals who cannot behave properly because they were not socialized when young.
But for humans it's more involved. We are essentially talking about a certain common cultural fabric and the problem is that if you don't fit that fabric, you will be at a disadvantage. Making friends, dating, interacting with coworkers - there's a critical window to keep your kids from turning into someone the average person would perceive as "weird". And the behaviors are subtle. Things like posture, voice tone, word pronunciation - I think "nerds" evoke a certain uncanny valley response in some people. That's a risk of homeschooling.
It may not be ideal but human nature is such that humans typically expect, on average, some degree of conformity. If personality and behavior are multidimensional vector, we're talking about an emergent clustering force, transmitted by socialization.
Edit: I speak from the experience of having spent years trying to catch up.
It's not just easing them in to socialization... it's giving them wrong socialization. We're all hot-to-trot about how important teachers are for all sorts of other skills both soft & hard, but it's apparently OK for kindergarteners to teach other kindergarteners how to socialize and it'll all be just the same as if they get good socialization the historically-normal way.