So, the question is really, "if it isn't real-estate developers trying to keep rents high, who has the money and therefore power to convince the government to adopt regulations preventing apartments?"
But you're conflating the two types of places: most places aren't "high" demand, and those that do have super inflated costs also have processes by which developers are regularly stopped. California in particular has this problem.
In fact reading your post really made it clear to me the dichotomy between places that welcome new people and have governments that try to serve the people (by letting developers build), and places that try to enrich current residents by blocking out newcomers and thereby forcing demand to be far higher than supply.
If you ignore "those cities not a part of a larger city's metropolitan area," it's (6,000/km^2) between New York City (10,000/km^2) and Boston (5,000/km^2). It's approximately the same density as Queens.
https://en.wikipedia.org/wiki/List_of_European_Union_cities_...
It's much less than Paris (21,000/km^2), Athens, Barcelona, and a bunch of other European cities. More than Dublin (5,000/km^2), if that helps.
https://en.wikipedia.org/wiki/List_of_cities_by_population_d...
Got nothing on Manilla, Mumbai (holy shit, Mumbai!), or Kolkatta.
But what the hell's up with Guttenberg, NJ?
The answer is primary that regulation by local governments prevents them from doing that thereby keeping real estate prices high.