At what point do you just call that illegal cartel behavior?

When they've captured the local government, you don't.

Middle-class NIMBY homeowners have literally zero meaningful input into zoning and permitting processes. At the absolute peak of their power they can show up to public hearings and whinge into a microphone for 3 minutes, an activity that planning boards and city councils around the country are perfectly happy to ignore.

If city councils and planning boards ignore them, they promptly get voted out of office. The NIMBYs are more reliable voters than anyone else.

That's a heartwarming narrative that bears no resemblance to observed reality. Frequently these positions are either totally uncontested or a straight party line vote by individuals who have literally no idea who or what they're voting for beyond party affiliation. What actually happens: council ignores the shit out of the noise coming from individuals that don't actually fund their campaigns, their affiliated party, or both. Entire projects are reframed as notionally beneficial to the community at large, citing some combination of bullshit claims about economic benefits and inclusivity, something something gentrification bad, etc, and the larger voting populace shrugs and goes on with their day oblivious. Worst case scenario a seat or two get sacrificed and the party apparatus advances another foot soldier to fill in the now vacated seat, with identical results.