Two years ago I did something similar to what you've done, ie drop sugar, but I never stopped eating pasta, bread or white potatoes (although I dropped considerably my intake of bread), and I still did lose some weight. I did not increase my exercise levels significantly.

I have wrote something about my experience at my blog: http://joaoventura.net/blog/2016/chart-diet/