I think incremental training should be possible. When next model is much smaller and as an input takes latent state from the bigger model. Then training will be faster, those smaller models can be chained.

Another option is to have small re-trainable areas within the big model.

Of course ideally model should be able to work with some sort of "knowledge base", which can be updated daily. As a main, or additional data source.

As for me I'm looking for ways to increase productivity. Have ChatGPT Plus, and in waiting lists for GPT-4 and Bing. Neither solves big problems, but usually there are many small things which can be done faster. I would rather offload them.

It won’t replace google search because ultimately it doesn’t have a point of view, let alone a dedicated page with a variety of functions (although plugins eat somewhat at this). For example, pick a random article written for The Atlantic — we are getting human prose from a specific author with a specific writing style about a specific subject that fits the zeitgeist of the day with a specific length suitable for what’s being communicated. That is something that is both enjoyable and informative, with style. LLMs are great at summarizing and synthesizing, but it’s not the same thing.

But perhaps the LLM can eventually point me towards this to then learn more; somewhat what Bing is already doing but it’s more of a footnote than something upfront (e.g. “you might want to read an excellent write up in The Atlantic about decriminalizing drugs”).