There's more than one kind of fitness. Bulking and cutting optimizes for maximum power-to-weight ratio. I won't deny that there's an element of vanity to it, as there is with any effort to be fit, but that doesn't make it "not fitness".

To borrow your phrasing, someone who follows GP's program instead of yours might not swim as well as you, but they're probably a hell of a lot stronger.