I control my calorie intake using published calorie counts (for mass market stuff), and a kitchen scale for anything I make myself. I also record every single thing in a diary. I eat a lot of crap (breakfast is either a hot dog or a bag of mustard pretzel bits), but my weight loss is exactly what I'd expect given my measured calorie deficit. If anything, its easier to measure your calorie consumption if you don't cook at home, because many mass market places now publish nutrition info, and they have a strong commercial incentive to maintain uniform portion sizes.
I did the same as you in terms of measuring calories, but mostly using scales or calorie counts on packets as I was cooking using raw ingredients. I recorded everything in a diary, using paper initially and then myfitnesspal app. I would also "cross-check" (e.g., after finishing a bottle of olive oil I'd look back at how much I'd logged.) I do think there's space for error and variation - e.g., a vegetable that's sweeter than average, or trying to judge how much fat (and therefore calories) is left on a piece of meat. That last one is the toughest because it can make a huge difference.
Probably some combination of these factors explains why you need to "ear FAR less than you should need to" based on how the math works out. I still think counting calories can be useful, but only in relative terms, not absolute.
In contrast, Indian milk tastes different when you buy from different vendors. You also have a choice between cow and buffalo milk.
I packed on an extra 20-30 pounds working in a catered, snack-heavy silicon valley company. I decided to change my diet completely. I slowly cut out all processed food from my diet, and then all snacks, so that everything that I ate was homemade by myself, typically "healthy" (similar to mediterranean diet), with no sugar, no juice, lots of vegetables and fish, some fruit, and relatively portion-controlled (eyeballed.) I lost 8 pounds but then plateau-ed - I continued to be overweight. I also exercised 4 times a week.
I started monitoring calories for a couple months and realized I was eating at a slight "calorie deficit" - I should have been losing a pound every week or two but I wasn't. Only when I cut my calories and STRICTLY portion-controlled my healthy food (to a calorie deficit where I should have been losing 2-3 pounds a week), did I finally start losing 2/3 pound a week. I'm proud to say I'm now a healthy weight but I eat FAR less and exercise FAR more than I should "need" to.
The journey really helped me realize there's something wrong with our food (not just the processed food but the ingredients themselves) and that there are some environmental factors involved (something in the water? the environment? pesticides? I really don't know. I'm not sure what it is.)