I have never seen a vegan who is what I would consider "fit" - and my standard is not necessarily an elite athlete, but, let's say, "able to handle themselves in a bar fight". Frankly a lot of them look rather sickly.

I have however met a lot of people who are Paleo or Carnivore who are in incredible shape.

I'm inclined to believe that at least some of this is cultural - Paleo/Carnivore folks tend to be more strict and disciplined about their fitness anyway, and do different workouts - and some of it is indeed due to diet.