My kids are not shy, and math-wise they're ahead of where I was at the same age. I went on to get a math degree from MIT. Your story is the first time I've heard of a Montessori school being so unstructured. Do you know if it was accredited?
I don't know, sorry.. I am sure that it was the Montessori system though. They even had pictures of Dr. Montessori hanging on the walls. I haven't really looked back into that time in my life, 'til now. I'm not upset, I turned out fine, I just personally feel that in a bigger school I could have become social more quickly by interacting with a wider variety of other kids.
As with everything, different implementors may be better/worse at the same thing. I've certainly seen that with my girl (who was in Montessori as a youngster).
Also, no one approach is perfect for everybody. As frightening as it may sounds, some people need more "structure" than others.
My final observation is that my Montessori school was so small! I was a shy kid, and going to a small school from K-3rd grade, then another small school from 4th-8th grade, did not help me to become a social person one bit. If you do something against the group, everyone will hate you, resulting in more shyness. Even at third grade.
You probably should evaluate your child. Is s/he shy? If so, it may be a benefit to send them to a larger school. I'm not saying that it will cure them of shyness, but it may help.
I'll totally respect any decision you make though :)