I believe these are pretty different scenarios. There's an ongoing, massive shortage of apartments in Berlin, and bots here often aren't bypassing "work put into adverts and apartment hunting sites"—they're trying to get an application in at all before the listing is flooded with hundreds of applications and disabled. The content of the application has nothing to do with it.
(Of course, this just leads to an arms race of apartment-application-bots, but this is the situation the city's found itself in. Same goes for registration appointments, residence permit applications, and so on.)
The bots are not a solution, they're table stakes. Withholding that information would not do anyone any justice.
For the readers, it also strips the information of a lot of nuance. I obsess over word choice for a reason. Then there are the footnotes, the citations, and the purpose-built tools embedded in the content.
As I said, I also lose access to the community and its feedback and encouragement.
That's now, before tech companies start throwing ads on top of the information it hoovered from my website. I'm glad that for a while, destroying the independent web will create value for the shareholders.
The premise behind libraries being public works rather than, say, private collections is that society as a whole is benefitting from this information being free and creating that information as well.
Here you are limited by the whims of a private actor which can be one way one day, and another way when VC money runs out or when it has a "fiduciary responsibility to shareholders" which is the inherent conflict the original poster is sharing.
Having all of the worlds information being parsed by private actors or non-profits who may become private at some time is a problem and the anthesis of "information wants to be free"
Still, it's a bit ironic that your own website advises people to use bots to get apartments (thus bypassing the work put into the adverts and apartment hunting sites), whilst also decrying bots who would bypass your site.
More generally perhaps there's a market for a Steam-like walled garden but for "old web" style sites. Crawlers would be controlled via technical means, micropayments would be built in, and use of AI would be strictly controlled. For instance maybe the garden uses AI to build link directories like DMOZ used to be, but the information itself isn't trained on so you're guaranteed to have organic readers.