I've never met someone doing pro-athlete levels of exercise that doesn't stick to some sort of regimented diet. Have you? And even if you do meet those folks, chances are they simply don't eat as much as before due to spending more time, well, exercising, and it makes for a caloric deficit, albeit accidental.

And that's the thing: Once you start changing your diet - the stuff you eat or the timing of your food - to do the exercise, you can't really say it is the exercise affecting your weight.