No a lot of experience in the subject but I believe that, unlike RF, if you need more OSNR you can always inject a lot of power in the fiber (I've heard about 5-10 watts lasers for long-haul data links).
Also, if you need to go farther away, you can always increase the FEC overhead.
First, fiber and just about any optical material have a nonlinear index. So as you increase the power into the fiber, the light modifies the local index. This creates a kind of noise. A wavelength channel can do this to itself even. When you have lots of channels on the fiber, there is a lot of modulation of the refractive index by all those channels. This is call cross-phase modulation (XPM). You can think of XPM as a phase noise. As you increase the power in the fiber, there's a point where the XPM "noise" increases faster than the improvement in OSNR. So you have to keep the launch power low to keep the nonlinear penalty low. This limits our launch power and ultimately the OSNR we can achieve. Generally we try to operate near the peak of this curve.
Also we're not there yet, but we're pretty close and it might not make sense to upgrade further, so some say that in the next few years we'll stop seeing data prices go down.
64-QAM does not go very far and requires very low linewidth lasers. The OSNR requirements for 64-QAM at 32GBaud means that you basically have no reach (distance). So makes a nice announcement but it's not a very practical modulation format. Any nonlinearity and 64-QAM starts cycle slipping like crazy.