I had a hard time with the beltway in Virginia/D.C. (I drove it twice and went the wrong way both times) and Denver/Boulder. I might hate them equally if I had to live in those places. I lived in Houston for 7 years and Austin for 6, and Houston has the nicest roads and signage of any city I've ever visited. Austin gets a lot of complaints, but I never had much trouble there.
I'm dysgraphic, and the directional component has the most noticeable impact on my day to day. If I have the option to turn and have no signs to tell me which way is which, I will choose the wrong direction at least 50% of the time. I occasionally get lost in parking lots (seriously, the big shopping center at Showers/San Antonio in Mountain View; I've been lost in that lot several times).
So, maybe for some folks California is fine. For me it is a seriously hostile driving environment. Luckily, I don't drive much.
I don't get distressed easily, and certainly not historically by the simple act of driving, but driving in California is extremely stressful for me. There are at least a few thousand locations in California where some guerilla public service would be a great help. I just get angry every time I drive here, because I'm always lost, always having to turn around because I missed my exit or turn, and even when I'm in the right place I can't tell because the signs suck so much.