Higher capacity optical storage is under development (see https://foliophotonics.com/ for example). My guess is that it's just hard to beat tape or justify upgrading/modifying existing optical disk readers.

The selling point now isn't that is a temporary storage solution that happens to last long like CDs and DVDs. I'm sure you have to make a pretty good case to get a customer to switch their high reliability data storage to a novel format.

I do hope it can eventually come back though.