It is a rare minority of the time in your life that you only interact with peers of your own age.
One of the touted advantages of alternative schooling is that the "students" have more natural relationships with peers of a wider variety of ages, better preparing them for the rest of their lives.
Don't you think you would have met other people wherever else you would have been away from high school?
Sure if you move on to college you will go and meet other people but you may be more intimidated as they will be 3-4 years older than you and you have missed the social style that school offers.
I have run into a few people who were home schooled and this is apparent as they usually do not understand how both the social and classroom aspects work: these people were especially hard to deal with in class as they would interrupt the professors often with useless questions at inopportune times that hurt the flow of information and in exchange move the entire class off track.
That was me. I firmly believe that middle school and high school did more damage than good for me, educationally. The only thing I really learned was how to get the A grade while doing as little work as possible. I could have learned and accomplished so much more in that time, and been better prepared for college.
However, I have a hard time asserting that I would have been better off skipping high school, overall. Education isn't the only thing you get out of high school. You learn how to handle yourself in a variety of social situations and with a variety of other people. High school might have handicapped my education, but skipping it would have handicapped my social skills.