I haven't had this experience at all. We have my number registered everywhere because my wife works at a lab and often can't pick up. I've never had anyone calling about the kids seem surprised or ask to talk to the mother.

Edit: actually, those places usually ask for a list of name and number of the parents/guardians. Maybe the number you pick up on was registered as being that of the mother?