Fiber has been used for over twenty years to connect storage systems to compute resources. Recently, Ethernet over fiber is becoming more common.

As @talidayo points out, the fiber connections are unlikely to be saturated. What hurts distributed systems is the overhead of message sending, the packing and unpacking of transmission data structures. As a rudimentary rule, individual nodes / services should spend only a small fraction of the resources and time for marshaling data and the rest on useful computation.