It was in reference to: The land needs access to water and electricity, not to mention roads, or it's a nonstarter, especially if you want to build a bunch of homes on the land.

(The comment may indeed be true--sort of by definition to at least some degree--for new construction, although getting that access can be pretty straightforward, to the degree anything involving construction is straightforward. It's more that housing prices are an issue in popular places where a lot of people want to live. And, yes, vibrant college towns fall into that category although not to the same degree as elite cities.)

Fair enough, it would be possible to scale up smaller towns, much smaller than Madison. However, I'm not sure that's practical, at least to the extent required to meet the housing shortage, for several reasons. First, the reason you already mentioned: "They may or may not have good local jobs." Second, there may not be enough home builders available in smaller towns. The builders themselves have to come from somewhere, and they're likely to be somewhat scarce the smaller and more remote the town.

I just don't think the conspiracy that the OP postulated, "We all want our properties to increase in value so we collectively conspire to suppress supply" is a very plausible explanation when applied in general to the entire country. For example, again in the Madison area, most of the home building is occurring not in the city proper but rather out in the suburbs and exurbs, which are independently governed. Indeed, in a fast-growing suburb, the property values can actually increase as homes are built and the town becomes a more desirable place to live.