This will be true as long as they are crazy expensive to produce. Eventually the cost of training a base model could drop to the range where crowdfunding could do it.

Or alternately someone could make a major advance in distributed training and we could all contribute cycles in a distributed effort like Folding@Home. As it stands training requires far too much bandwidth for synchronization and moving model data around. Some approach to sharding training would have to be discovered. It’s an open problem area.

Neural networks are very parallelizable and training is stochastic so my intuition is that it should be possible. Even if it were less efficient than synchronous training you could make up for that by harnessing 100X the compute from a huge crowd.

Platforms and rightsholders know what they're sitting on now, and I worry that the datasets required to train sufficient models in the future will also be hoarded as IP.

It's one thing to train on Common Crawl in 2023, but what about when you have to shell out millions of dollars just for access to data sets to train on in the future? Same thing with human reinforcement. The customers for both are willing to pay much more than a crowdfunding campaign would.

Training is expensive now, but data sets can be expensive in the future.

Collecting training data is actually a perfect for crowdsourcing. Images/videos are easier than text, and text is easier than high-quality text, but all are doable.

Only so far as you trust your crowd. It only takes one bad actor to poison an entire data set with a few pieces of copyrighted samples, and now the entire model is subject to potential future litigation.

Wikipedia somehow handles this okay. But yes, there will be work to do to address the issue.

How is it different from stock images? People could sell not their work on the stock sites, and it is pretty hard to detect. Or impossible to detect, if you ask some AI model to repaint the image first, before putting it into the stock.

I feel like the amount of training data needed to compete with commercial models would require a large amount of automation to collect, and it's that same automated collection that platforms recently decided to gatekeep against.

have we exhausted the ways to train models already though? Feeding common crawl into a more advanced training program could result in a better model compared to today

Yes, but as time goes on, new data is created that won't be in Common Crawl. Many of the troves of data it archived from in the past are now behind expensive paywalls that are priced with their multibillion dollar clients' access in mind.

If the dataset contains text from 2022, things that happen in 2023 and later won't be in it. The model will only get you so far, and new data, events, concepts, discoveries, etc will be absent.

If we trained models on all text generated up until 1900, for example, you could get it to produce some impressive results if just generating text is the goal. If the goal is to build something that imitates a more general AI, it wouldn't "know" about antibiotic treatments for common illnesses, modern vaccines, either World War, powered flight, transistors, computers, etc. It would only be useful for so much.

Yeah but that isn’t t pay - per - operation

I mean I guess my electricity provider gets paid per compute.

You can write software that is pay per use, and you can do the same with local models.