Remote work from home, where you are ensconced in your isolation box, is gross, too; as a consultant, I have a home office and I spend most of my time out of it.
(Mind, I have considered buying something in a rural area--but that's effectively a retreat, not a home. Homes are a by-and-large a pipe dream for people who are under 30 right now unless they want crushing commutes or nothingvilles.)
I actually love going on long walks, so the thought of walking a half hour to buy groceries if I don't buy a car seems ok.
I've spent the last decade or so in San Francisco, and none of the local scenes really attract me. There used to be some cool artists around, but a lot of them got priced out. Maybe some other city has a more thriving cultural life, but San Francisco's isn't doing anything for me. I just have to figure out where to move.
Well, I hope that we both manage to figure out a way of life that is good for us.
What you're describing in terms of collapsibility, zero space, and cultural lifelessness is why I have avoided San Francisco like the plague, but San Francisco isn't like most urban areas--when asked why I refused a job with a startup out there I responded with something very similar to what you're saying here. =) I live in the Boston area; the city I live in is on the subway line, but isn't Boston proper.
I live on the top floor of an apartment building; my place is about 1200 square feet. I regularly sleep an off-kilter schedule. My neighbors like me, and me being up at 3AM doesn't seem to bother the downstairs neighbors seeing as how we hang out once a week or so.
Rural areas are fine if that's your thing, don't get me wrong. But there's a lot that you can't get in those areas (and I've spent plenty of time in them) at all.
I too work from home, and the difference is so stark, I can't imagine ever living in a less dense neighborhood again. I haven't gotten in my car in a week. Not going back.
The thing that prevents new construction in many areas is laws designed to "preserve the character of the neighborhood". Homeowners/investors have an incentive to push for these laws to keep housing prices high. However, if they actually own a diversified asset that can grow with new construction by adding new properties, then that incentive is reduced.