I think basal metabolic rate can vary significantly. This might explain what you're observing in a more biologically plausible way. My (imperfect) understanding is that it's partly genetics and partly in your control via strength training.
But I think you're doing yourself a disservice if you're discounting too easily the possibility that you're not comparing your eating and exercise with theirs accurately. How confident are you that you're actually eating no more calories than they are in a typical week? You might see them eat a huge amount when you're together, but maybe that's their "cheat" day, maybe they don't snack or drink alcohol as much as you do, etc. I think a lot of people are fooling themselves when comparing to others.
[1] not a recommended weight loss plan; side effects can include stress fractures, scurvy, IBS, and on and on.
And yes, I do exercise a fair amount and all that, but I also seem to be quite efficient at extracting energy from food. I tend to joke that at least I'll survive longer after the apocalypse.