To use your same metaphor, there are people going around speaking as if height and total hours of training are the only things that matter for basketball skill.
Ultimately you set a caloric target with a carefully constructed diet and watch the long term weight slope over months. That's the science of it. Even high fat ad lib diets ultimately resulted in caloric input restriction, stronger than other diets they tried. (see Cambridge study)
Agreed, but I'd characterize all that as focusing on the "Calories In" part. That's complicated enough itself, and avoids implying that the entire situation is some trivial function.
There are at least two categories of issues that make calories in not so simple: eating disorders and malabsorption disorders.
There are so many separate ways for calories to go "out", too.
So it gets used in this unfair double-standard way, where managing weight is "just CICO", as if it's a handful of inputs into a one-line equation, but if someone argues against that, saying it's more complex, then the natural response is "well actually CICO includes every single complication you've mentioned, so you're wrong to object".