I think what I mean by "bad" is that removing sugar (or at least food with added sugar) is like a low-hanging fruit on the road to a better diet.

White bread is not too far away from sugar, like alcohol, actually. Replacing white bread with something full grain would also be something "good" for your diet – although in some places it's really hard to find full grain bread that doesn't taste like parchment, so maybe a not-so-low-hanging fruit ;)

Finally, I think I am compounding the refined sugar and the sugar lobby itself, who has been pushing for decades that what is bad for health is fat, whereas study after study proved otherwise.