It all comes down to energy. Each time energy gets converted as it goes through the food chain, most of it is lost. We simply can't afford to eat things that ate things that ate things that ate things; there's not enough energy left after all that.