Plus drugs are one of the very most heavily regulated "free markets" in the US ... and as others have noted, there are laws like the Bayh-Dole Act (https://en.wikipedia.org/wiki/Bayh%E2%80%93Dole_Act) which can make the process much more of a government-non-profit-corporate partnership. There is also a "Generating Antibiotic Incentives Now" (GAIN) in effect, with an bipartisan Antibiotic Development to Advance Patient Treatment in the works.
So, no, we don't "just take whatever the free market finds profitable to provide", where "profitable" has "the visible foot" of the government strongly weighting one side of the balance scale. Not to mention very strong medical policy to restrict new novel antibiotics like this one to the cases where they're really needed.
There have been lots of proposals to overcome this. My personal favourite is each country agrees to pay a straight cash bonus for each new antibiotic developed with a different valued bonus in proportion to the need. The drug companies would then not need to worry if the antibiotic was used or not and we would have on tap new antibiotics to use as resistance arises.
The reason we are having trouble finding new classes of antibiotics is that we are already have chemicals that target most of the interacting and essential cellular modules in bacteria. If you already are targeting 90% of all the possible targets you will find it very hard to discover any new classes of antibiotics.
Of course the solution is not to worry too much about finding new classes and just keep tweaking the current antibiotics to keep ahead of the bacteria. The only problem with this strategy is that the economics of antibiotic development is totally broken so the pharmaceutical companies have in the main stopped spending money on finding new antibiotics. What we need to work on is a new way of paying for new antibiotic development.