A vegetarian diet often has significantly less protein (important if you're trying to build or maintain muscle mass) and less of some trace nutrients (zinc, iron, vitamin B12) than a diet including meat. It's possible to make up the differences and there are health benefits if you do, but it's definitely more work.
It is limiting your choices in restaurants obviously, but suppose you live in some vegetarian community. Is this community limiting its potential? While if you live in a community that exclusively hops around on one leg, the answer is obviously yes, for the first example I'm not sure.