Which private party decides that?

The municipal council approves all building plans. Just because you own the land doesn’t mean you have the right to build whatever you want on it without approval.

At least, that’s true in at least one state in the USA.

The neighbor or utility providers have a say if there's an easement, e.g. you can't build as to block their right-of-way access for maintenance purposes, despite your ownership over the land.

Also the developers and architectural controls can make decisions affecting what you build (down to the color you paint outside), before they turn things over to the city.

All subject to local laws of course, so may vary city to city.

Home Owners Association, Neighborhood Association, Historical Society, etc