It's definitely not just about muscle mass. But you should compare yourself to yourself only when it comes to self-improvement, since you can't borrow someone else's metabolism. Having more muscle mass is better than not having it for both health and appearance.

When we're talking about slowly gaining weight over time as the OP did, even if muscle only burns a little more energy than fat that could be enough to tip the balance.