I would personally remove simple carbohydrates instead, if I wanted to lose weight. But you need to do what works for you.