That's my experience anyway. I spend my summers in a friend's farm and they have a small army of chickens, a plattoon of goats and in different times, a couple of pigs or so. We get fresh eggs and fresh goat's milk, but all the meat we eat is bought from outside the farm. It seems to be a very good deal actually, which was surprising to me when I first realised it, because of course the chickens and goats have to be fed. Then again- they often eat what we do (or rather, our leftovers).
Our ancestors generally ate far less meat.
They could also feed smaller animals like poultry on scraps of food, which doesn't really scale to feeding the billions of animals we currently raise for food.
I don't argue that we should continue todays consumption, only that going all vegan might not be a good idea.
Reducing consumption seems like a really good idea to me for several reasons. As does small scale poultry and animal farming (raising a pig or some poultry on scraps instead of throwing it away should be a good idea today as well if one could get away with it.)
One easy thought experiment that helped me understand: given how extremely poor our ancestors where and how hard they worked to make ends meet there must have been some advantage to keeping animals.
Otherwise they wouldn't have had them, because they are sometimes a real hassle and gathering food for them is time consuming I can tell you.
So maybe your idea would work in a more temperate climate. And maybe it would provide extra food for people further north, but it would mean a lot of land that cannot be used for anything but pastures would go out of use.