Regardless of added sugars, all bread is about 50% carbohydrates. As far as your body is concerned the only real difference between straight sugar and more complex carbohydrates is how long it takes to metabolize. Complex carbohydrate are less of a sudden jump to your blood sugar, but about the same net effect overall.

You’re better off limiting carbohydrates in general than just refined sugar.