Campbell also has a nice downtown; while it's not San Jose, it's surrounded by San Jose and is basically the same metro and is VTA-accessible. There's also Saratoga and Los Gatos, but those are further out.
Downtown, I'll grant you, though. San Pedro Square can be OK, and the area nearer the University has some quirky stuff, but it has a ways to go. It's gotten somewhat better over the years, though, and I think BART coming to the area may accelerate that a bit since it can actually be a destination.
But here's one thing I've found: People who really want to have fun find a way. There are small pockets all over the bay. SJ downtown is a little lame but it is getting better
You're overstating the issue. It's not New York or San Francisco, for sure, and we sure do need better rapid transit but you're definitely not an hour and a half from everything cool. You're more like half an hour or so. You just need to look for it.
Sure there are disproportionately more workaholics in the area. There are also disproportionately fewer people who work 9-5 then go home and veg out on their couch. To me, the lifestyle here is rich and amazing. Where I grew up, which was in an equally large and cosmopolitan city, it's too cold or too hot for 4-6 months of the year to do much outdoors, salaries weren't high enough for people to routinely go to expensive restaurants or travel, and the geography is nowhere near as beautiful or interesting.
Maybe you should look inward more and take charge of your life; it may be that you're hanging around people who focus entirely on work, and so your perception is skewed.