Cutting down on calories necessarily means cutting down on fat, carbs, or both (technically protein as well, but protein doesn’t generally seem to be the issue.)

If you were to follow an explicitly low fat diet, it would be very difficult to eat junk food as well. Almost all junk foods are high in both fat and carbs at the same time.

You are right about refined carbs. Most public health organizations advise limiting one’s consumption of those.