I eat lots of carbs, always have... yet I don't seem to gain weight. I generally feel like, and am regarded by people who know me, as someone who "eats a ton" yet I have never struggled to lose weight.
The X food causes obesity theory is just too simple IMO, that or I have a tape worm or some kind of undiagnosed disorder that results in me maintaining a pretty consistent BMI despite eating whatever and whenever I want.
Cutting down on calories necessarily means cutting down on fat, carbs, or both (technically protein as well, but protein doesn’t generally seem to be the issue.)
If you were to follow an explicitly low fat diet, it would be very difficult to eat junk food as well. Almost all junk foods are high in both fat and carbs at the same time.
You are right about refined carbs. Most public health organizations advise limiting one’s consumption of those.
In my personal anecdotal experience, my weight gain and loss correlates to my caloric intake. When I've explicitly calorie counted, foods with refined carbohydrates are typically what blows up my count. When I've done low carb high fat and protein diets, I've found it difficult to eat too much, to the point of coming up several hundred calories short per day.