It pretty much comes down to weather.

Importing water is easier than importing sun and heat.

But who knows, that might change:

https://www.vox.com/energy-and-environment/2017/11/8/1661171...

...until said water runs out, then you’re hot, sunny, and dry.

Climate is the big one, land and access to water (I know this seems weird for California but they do have it). You could grow in West TX, southern NM and AZ but the water logistics are not there. You see the same crops in FL, but they have less land to support the same kind of agriculture infrastructure as they do in California. The labor situation is a symptom of the above factors, by its very nature its mobile. If you had the farms to support it, the labor would follow.

Availability of cheap agricultural labor?