I agree with you overall, but some vegetables were found in a recent study to be more efficiently grown at home. Also, a lot of people already have turfgrass farms and converting those to veggie farms would be a huge net win. Finally, if you think that people have a fairly static amount of time/energy to spend on hobbies, hobby farming would likely be a net benefit because some hobbies have much greater negative externalities.