This street in Chicago (http://www.iroquoistheater.com/pics/madison-street-chicago-i...) is typical of most cities of that time, be it Cleveland, Cincinnati, Buffalo, or New York City. The effects of the automobile were only just starting to be felt.
Cities had to be walkable because apart from trolleys or horse-drawn carriages there wasn't much else in the way of options. Cycling was an option for the brave as I can only imagine how much horse crap there was in the streets back then.
Chicago has the advantage of plenty of land as opposed to New York or SF.
There are massive sprawling suburbs, but I get the impression people move there when they have kids mostly due to better schools, not being priced out or leaving for safety.
Of course, Chicago has the most murders in the US (but still about half the murder rate per capita of New Orleans or Detroit), bitterly cold winters, and far from any mountains, but nowhere is perfect (where being affordable is part of being perfect).
https://en.wikipedia.org/wiki/List_of_tallest_buildings_in_W....
NW is the largest quadrant, but I'd hardly call the other quadrants "relatively empty." There are active neighborhoods and lots of development all across the city.
A lot of people would like to live in cities like SF but it's been illegal to build them since the early post war era, so there's a big supply and demand imbalance.
As in, despite Brooklyn's relatively functional nature, its density gains are counterbalanced by the suburbs of NYC on the east, north, and west.
The grandparent post implied that it's possible to have an environment that doesn't include high-rises and excessive sprawl, and I believe Brooklyn's heavy integration in the NYC metropolitan area makes it so that it doesn't satisfy this constraint -- it just moves the 'problem parts' immediately outside of its borough limits, which is kind-of how we ended up with suburban sprawl in the first place.
HTH.
Madison, however, on first blush seems like an excellent example, in that it's a large (~250k) city ranked highly on most metrics, and has a mid-rise downtown. But a look at a satellite image reveals that the city limits of Madison contain large areas of single-family detached houses, and mid- and high-rises are restricted to the Isthmus and areas immediately surrounding the University [2].
[1] https://en.wikipedia.org/wiki/Philadelphia#Crime
[2] https://www.google.com/maps/place/Madison,+WI/@43.0850975,-8...
Center city is pretty safe. So is South Street. Old City. Plenty of other neighborhoods too numerous to mention with this style of buildings. Yes, crime is relatively high in the city, but it's concentrated in certain areas (see: http://www.trulia.com/real_estate/Philadelphia-Pennsylvania/...). There's a reason the Badlands is called the Badlands and Center City isn't. Judging the whole city to be unsafe by its overall crime stats is ridiculous, unfair, and misleading.
Ultimately, NIMBYism and these sorts of height restrictions are a result of the city deciding that it doesn't want new people moving in past a certain point. I won't moralize that choice, but it is certainly different than the set of choices that led to a city like New York becoming so special.
One of the biggest differences in these types of cities is parking is very limited or located underground, the buildings push very close to the streets, there's no excess road or empty front yards, and the buildings are mixed retail/residential.
Those things make for consistently more livable environments. There's no long trek across an empty parking lot and a big lawn just to get from point A to point B.
The result is it doesn't feel nearly as dense as it actually is.
This is in contrast to Hong Kong, São Paulo, and other cities that achieve density through high-rise buildings. You're intimately aware you're in a big city when you're in the built-up cores of these.
Also, I think it's very unlikely that NYC would be where it is had they restricted buildings to 5-7 stories. As it is they don't have enough density.
You can have human-scale development, buildings 5-7 stories tall, moderate density that promotes local commerce, safe environments for children and other vulnerable groups, easy access to amenities, and much more.