We grow a lot of human-edible food for the sole purpose to feed it to livestock, who then spend most of those calories on existing and put a small portion into body mass that we eventually eat.

Sure, that stuff isn't of the same quality as food grown for human consumption, but putting livestock on a diet and diverting some of their food to human consumption would more than cover any shortfall from the missing meat