The building codes aren't the way they are because you need different rules for smaller buildings. It's because existing residents don't want smaller buildings. If you can build smaller buildings then people will, and then you have a bunch of new students in your school district whose education is paid for from local property taxes, only their parents aren't paying a proportionate share because they have a smaller house with a lower property value. Which means the existing residents would have to make up the difference, which they don't want to do, so they have the building code prohibit building smaller homes.
It's basically regulatory capture by existing homeowners of larger homes.
Zoning can explicitly set lot size minima or density maxima.
Building codes with specifications for door and corridor widths, accessibility ramps, minimum room sizes, bedroom counts, bath-bed ratios, height limits, basement exclusions, attached vs detached dwellings, "in-law" units, with or without seperate entrances and/or cooking facilities, etc., all indirectly efect housing size and density.
Local jurisdictions will usually have addenda to the code, for local fire requirements, etc., but a lot of that flexibility is baked into the standards themselves. In here there may be regulatory capture from local builders, and at higher levels (state-wide) there may be some capture from larger construction companies, but there's very little in the way of a feedback loop to allow homeowners to significantly change the building code.
Rather than debate in abstracts, I'd be curious if there is a particular section of the code that you feel is a result of this sort of capture. The IBC is available for browsing online [1], and Chapter 5 has the base restrictions on building size, with some notes on justification (largely about at what size you require additional fire suppression, which is a cost that most builders want to avoid in residential settings).