The speed of light in an optical fibre is about 35% slower than in vacuum -- it depends on the index of refraction.

https://en.wikipedia.org/wiki/Speed_of_light

The same is true for light traveling through water, air, and so forth.

Is it possible to build a multi-mode "fiber" with an air core so that a significant part of the light's path is through air? The outer sheath made of optically dense material would only be used for internal reflection to "guide" the beam...

No. The core has to be denser than the outside for the guiding to work; if you made a "fiber" like you describe the signal would just travel through the sheath - if it was heading towards the core it would internally reflect to stay within the sheath.

My understanding of it is that when light passes through a medium, it does not do so in a straight line. So, while the actual speed of light remains constant, it takes longer to get from one end to the other.

To be precise, light can travel through a medium in a straight line, but it bends when it passes from one material to another. The light bounces off the inner walls of the fiber in a zig-zag fashion as it propagates, so it cannot possibly travel at the speed of light. It depends on the index of refraction as mentioned.

No, the speed of light decreases in a medium – in fact, the proportional decrease in speed is the refractive index of the material.

So much for it being constant huh?