Delays in boarding are slow because of people sitting next to each other who are not traveling together. If you have a minor with a window seat and you are sitting next to them, there's is no reason you can't board at the same time; you're not going to be getting up to let someone in after you've sat down.

One abuse I don't like but see often is the "we are traveling with an extended family of adults and one baby, so everyone gets to board 'early' due to the traveling-with-infant rule".