What you're missing is many people in modern society have decided that kids are solely the responsibility of their parents and no one should ever suffer any inconvenience or displeasure as a result of them. However slight it might be. This means anytime your kid decides to act out in any way in public, including occasionally make a sound above speaking level, you are disrupting other people's experiences and get dirty looks by these people.

Also, don't think to do anything when there are fewer crowds, because this is when these types of people tend to congregate the most - they like their empty, quiet, orderly spaces and your kids' presence is most definitely not welcome at this time.