Still, it's a bit ironic that your own website advises people to use bots to get apartments (thus bypassing the work put into the adverts and apartment hunting sites), whilst also decrying bots who would bypass your site.
More generally perhaps there's a market for a Steam-like walled garden but for "old web" style sites. Crawlers would be controlled via technical means, micropayments would be built in, and use of AI would be strictly controlled. For instance maybe the garden uses AI to build link directories like DMOZ used to be, but the information itself isn't trained on so you're guaranteed to have organic readers.
I believe these are pretty different scenarios. There's an ongoing, massive shortage of apartments in Berlin, and bots here often aren't bypassing "work put into adverts and apartment hunting sites"—they're trying to get an application in at all before the listing is flooded with hundreds of applications and disabled. The content of the application has nothing to do with it.
(Of course, this just leads to an arms race of apartment-application-bots, but this is the situation the city's found itself in. Same goes for registration appointments, residence permit applications, and so on.)
The bots are not a solution, they're table stakes. Withholding that information would not do anyone any justice.
For the readers, it also strips the information of a lot of nuance. I obsess over word choice for a reason. Then there are the footnotes, the citations, and the purpose-built tools embedded in the content.
As I said, I also lose access to the community and its feedback and encouragement.
That's now, before tech companies start throwing ads on top of the information it hoovered from my website. I'm glad that for a while, destroying the independent web will create value for the shareholders.
The premise behind libraries being public works rather than, say, private collections is that society as a whole is benefitting from this information being free and creating that information as well.
Here you are limited by the whims of a private actor which can be one way one day, and another way when VC money runs out or when it has a "fiduciary responsibility to shareholders" which is the inherent conflict the original poster is sharing.
Having all of the worlds information being parsed by private actors or non-profits who may become private at some time is a problem and the anthesis of "information wants to be free"
We are surviving, but we've changed our entire focus as the web we know is a dead man walking. AI bots are the new frontend for the web, and I am curious about what the web will look like in 2 to 10 years. I hope the AI companies start paying to access the data as they answer queries or construct dynamic frontends (but I doubt it).
I have many friends who ran great indie websites, and it has been sad to see that 90% of them are now frozen or closed.
I am curious what it looks like when you need a good product review in two or three years. Is YouTube the only trustworthy source by real people? For most products, the web is just going to be Astroturfed Reddit posts and Forbes AI-written junk that didn't even have the product to test.
I'm hoping paid MCP data sets might get there, then I could make my data available there, and hopefully make a $1 to $4 per thousand requests to the data set. If that replaced the loss of Google traffic, that would be amazing.
I'd love for 25% of my AI subscription to go toward that each month, i.e., I can pull "a list of all science fiction books the NYT recommended in the 1950s" and it gives NYT a few cents to go through their data...
I also run an online website that provides valuable information. However, we have a very different philosophy. The mission is very not not to "put new information on the internet." Rather, it is to provide a valuable and much needed service for people in our niche. While we do greatly benefit from SEO, we also work very hard to cultivate our audience, both in terms of finding new people, and nurturing our relationship with the existing ones. "Information on the internet" has zero inherent value without an audience. If you're not working to build that audience (outside of third party platforms), then it's going to be very tough in the long run, and you'll run into situations exactly like this.
FWIW, our SEO traffic has remained shockingly steady, though the search console has reported a dramatic increase in "impressions," likely due to featuring our website in AI generated summaries. We'll see if things change over the long-term, but for now, we're not changing any strategies based on this.
Google was the lesser evil, but I still fully expected to get killed by an algorithm update some day.
I vowed to ride this website into the ground if I have to. I believe in its value, even if I need a side job to support it.
The internet was a beautiful place
Or the credit. Or even just the feedback.
Even LLMs get at least some feedback.
Never used it, maybe someone can explain how it has evolved.
- Newsletters. You own your audience and your list is portable and interoperable.
- PWAs with push notifications. Be careful not to overwhelm users with too many messages. Make opt-out options easy to find.
- Door flyers with QR codes. A return to the basics that can still be effective.
- Social media. Still useful, but its reach and engagement seem to be declining.
My main mission is to put new information on the internet. It's harder to do this if AI destroys the economics of it. It's also harder without an audience who provides feedback and encouragement. Having all information mediated by two companies isn't just digging into my revenue, it's killing the fun.
The unfortunate side effect is that it made me focus more on business and less on serving my readers. I'm working on health insurance stuff when I would rather work on a citizenship guide. It's also sad to see all the nuance stripped from my carefully chosen words.
It sucks, honestly. I am being replaced by AI, but I am still supposed to go experience the real world and translate it into LLM training material. In the end, there still is a thinking, feeling human being doing the essential work. He's just not getting paid anymore.