The obvious downsides to glass bottles are weight and fragility.

I personally find them preferable to plastic. But aluminum is (roughly) as light as plastic, and it doesn't catastrophically fail in the way that glass does.