The whole line about "it's just calories" is precisely that it does not matter what you eat, it's all just calories. If you agree that it does matter what you eat, it is a fundamentally different position, and it means that, like I said, you are no longer defending the conventional wisdom, you are stipulating the correctness of the attackers, while still confusing yourself into the idea that the original position that you have now stipulated is inaccurate is still somehow true.