Your body temperature goes up and down, changing how much energy is spent by body. Your ability to perform goes up and down - you become sleepy and tired, your brain spends less energy and that is that (yes it has impact on your attention and performance). You cant stop it from happening. You may try to exercise the same and can easily become unable to.
Your body gets rid of calories in poo and piss, when sweating or not.
Your body can just turn off functions and generally malfunctions and you dont have direct control over it.
If we looked at total energy of your gasoline we'd need to consider mass to energy conversion as well... but your body can't do that either. Energy your body can metabolize is clearly a subset of total energy available. It's certainly an estimate and it's not perfect but it's useful.
This looks like a digestible introductory guide: https://themedicalbiochemistrypage.org/digestion-and-digesti...
And a brief description of how all that helps inform calorie counts on nutrition labels: https://en.m.wikipedia.org/wiki/Atwater_system
People make entire career studying these sort of processes. Over time the information is refined. All of this ultimately helps inform how we measure and quantify things like calorie counts on food labels. Again, not perfect, but the process works and improves over time. It's a complex macroscopic system relative to the underlying chemistry and physics but if we assume larger systems are built on, composed, and governed by principles of constituent systems then we have a way of slowly understanding these complex processes and using that knowledge to our advantage.
Just as with software systems we rely on abstract processes below us to talk about things. That doesn't mean the high level abstractions are ignoring the details, it's just that there's value in talking about things simplistically like "calories in" and "calories out" without people looking to take very literal interpretations and trying to convert total energy possible into kcals. Heck, this is why they're "Calories" and not "calories" and why "Cal" has even been proposed to help avoid this sort of confusion.
The assumption that body resembles simple closed physical model is absurd.
Food (calories in) is just a form of energy your body is capable of utilizing. Fat is just a physical medium to store energy for use in your body. For your body to move and perform actions (live), you require energy. All things considered at this scale, if you expend more energy than you consume, you'll ultimately either die (due to lack of energy) or use other forms of energy in your body (e.g. fat).
This is of course the 30,000 ft view and there's a lot of complex biology and chemistry going on but it's pretty straightforward if you believe the law of conservation of energy is correct and our bodies aren't doing some sort of nuclear physics to process energy and instead relying on various forms of chemical and physical energy.
As you point out, it's not so cut and dry though in practice. There's a time component here and our bodies get quite clever. We build up efficiency of certain tasks over time requiring less energy usage. We resist building muscle because it requires more energy to maintain. Some food's energy density isn't perceived very well by our bodies and we remain more or less hungry irrelevant of potential calories making it more difficult to maintain a calorie deficit/diet based solely on our bodies senses. Many nutritional information products don't accurately measure calories of food, sometimes making it difficult to use as a base metric (theres a lot of errors in these measurements, especially by small food producers/providers). We also have incredible difficulty trying to estimate how many calories our body is actually expending because it's a fairly complex system (the "calories out" part of the equation) although there are some figures out there to help you estimate these within some general ranges depending on factors like age, diet, fitness, activity and so on--it's not perfect but it's something to work with quantitatively.
The benefit of calorie counting though is that there are fairly basic physical and chemical principles at play that make it a useful tool in weight loss and management. Even if a bit variable (both the in and out) and sometimes inaccurately represented, it gives us something fairly quantitative to work with, measure, and change around to control our weight. We can measure it and change other variables in our lifestyles to, over time, manage weight. It works and the statement is valid but it's not simple to use by any means, there's a lot one needs to learn over time to better learn how to measure their in/out and overall body feeling based on these factors.