Real estate has very distributed ownership and none of them are motivated to form a cartel.

The only way they can maintain one is through legal force. Zoning laws are almost entirely that force; they're a way to establish a cartel of homeowners.

Cartel's can't just magically set prices.

They can withhold supply, and that might have an effect on market prices.

You can't pay the mortgage on a place you're not renting out though, so they mostly can't withhold supply.

It applies more to things like retired people living by themselves in a 3 bedroom house.

OK.

Though if they can't withhold supply, how are those cartels supposed to hold up prices, then?

They don't, it's not real.

(Except that they can prevent construction of new units through zoning.)

Ehm, what? Of course cartels can magically set prices. That's what a cartel is for: to make collusion easier.

Even OPEC can't magically set prices. Cartels make collusion easier, but you still have to do concrete steps to raise market prices.

OPEC does that via production cuts. Or rather tries to do that.