Residents do have the most influence and the most concentrated control. That's not quite the same as being the only authority, or having absolute influence however, and this is a correct allocation of power.

Residents don't and shouldn't expect to have the final say on who can live in their area. This is explicitly acknowledged by, for example, racial housing covenants being illegal these days.