Legislators seem to disagree with you and passed a bill to require relevant cities to zone for farmworker housing https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm... and there is another one proposed this year.
Everything I read attribute this to a lack of housing supply. https://www.mercurynews.com/2017/09/19/housing-affordability... and http://capitolweekly.net/crisis-affordable-housing-grips-cal...
For instance, Stockton leads California rent increases with a 10.4 percent increase in rental price between July 2016 and July 2017 and their home prices have increased 92% in the last 5 years. In a supply balanced market this would not happen as the landlors do not have leverage to increase rent this much, and developers/owners don't have the leverage to sell housing for this much more.
If they had suggested merely that housing costs are an issue for californians, I would not have argued.