Pants are warmer and I don't have to worry about how I sit (am female, and do not enjoy this bit about dresses). Biking is not an issue, nor is extra fabric going to get stuck in a conveyor belt.
I'm also pretty sure the togas would require a decent learning curve if you are wanting sleeves. I certainly want sleeves and a shirt that isn't in the way: Things like ponchos aren't good to cook in - they get in the food.
Togas would be outright dangerous when swimming and I can't imagine them being great for working on a car. They might be slightly cooler in the summer if you aren't using too much fabric.
And I'd argue that no matter what, you still have fashion. You'll see it in the way folks drape the toga fabric and their use of belts and things to hold it together. You'll see fashion in the color and/or print of fabric and whether or not you can keep a white one clean. Like with uniforms, you'll find more emphasis on body shape, hair, and other such things.
But I suppose we could all just let our hair grow out: women and men both should just stop shaving as well. It is, after all, fashion. And not do phsyical fitness more than what is required to be healthy and do your job - excess muscles are fashion, after all.
(Since that's a name I hadn't heard in a while, I had to check to see that they still existed)
gymnos = naked
Everything more complicated than that is fashion.