Actually, a lot of the perennial food crops we do or could eat are adapted to environments containing ungulates or other animals, and there are a number of nutrient problems and pathogen vectors that wouldn't be as much of an issue if the land was being judiciously (rotation) grazed.

What will end up being seen as crazy is locking them all up in a barn and burning fossil fuels to bring the food to them instead of them using their own legs, all for another 20% body mass and giant meat portions (think: steak, chicken breast).