I've been on this journey for over ten years now -- basically on the practice and pain path you describe. But it wasn't until I read Allen Carr's "Good Sugar Bad Sugar" book that I began to mentally relate to sugar and carbs like any other significant addictive drug like nicotine or heroin. The sugar and carbs are not just passive calories that you ingest -- they actively affect how your brain and body relate to food.
Once you make that mental switch, then it's much easier to drop and stay off them. I mean who would say it's okay to have heroin cheat days?
There are other ways to do this, but they require you to leave your world behind. For many, their environment is what built and maintains their unhealthy relationship to food.
For me, the day I left Texas was the day I started dropping pounds. Then, the day I left Utah for London was the day I dropped even more pounds. Then, the day I started running was the day I began approaching my lower limit (likely bordering into unhealthy body fat % on marathon race day).
The biggest challenge of you and the commenter above you was that you were swimming up stream. Hopping into a stream that leads away from fat behavior, instead of toward it can do much of the work for you.
I fully appreciate that's a luxury, but it's something people who have a choice should consider.
It's not about the diet - most diets actually work, if you can stick with them for the rest of your life, but that's a huge if. CICO or keto or IF or carnivore or whatever, doesn't matter. Permanent weight loss requires the much harder task of fundamentally changing your brain's relationship to food. The only way that happens is through practice and painful failure.
You do have control over your weight. But losing fat and maintaining a lean body will be one of the hardest things you ever do.