The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.

It will never completely replace Google search because no matter how large the model gets it will always have a training cut-off date in the past, and there will always be specific factual information that isn't a good fit for language models.

If you're looking for a LLM replacement for regular Google search then Bing and Bard are much more likely to fit the bill.

I think incremental training should be possible. When next model is much smaller and as an input takes latent state from the bigger model. Then training will be faster, those smaller models can be chained.

Another option is to have small re-trainable areas within the big model.

Of course ideally model should be able to work with some sort of "knowledge base", which can be updated daily. As a main, or additional data source.

As for me I'm looking for ways to increase productivity. Have ChatGPT Plus, and in waiting lists for GPT-4 and Bing. Neither solves big problems, but usually there are many small things which can be done faster. I would rather offload them.

It won’t replace google search because ultimately it doesn’t have a point of view, let alone a dedicated page with a variety of functions (although plugins eat somewhat at this). For example, pick a random article written for The Atlantic — we are getting human prose from a specific author with a specific writing style about a specific subject that fits the zeitgeist of the day with a specific length suitable for what’s being communicated. That is something that is both enjoyable and informative, with style. LLMs are great at summarizing and synthesizing, but it’s not the same thing.

But perhaps the LLM can eventually point me towards this to then learn more; somewhat what Bing is already doing but it’s more of a footnote than something upfront (e.g. “you might want to read an excellent write up in The Atlantic about decriminalizing drugs”).

Try https://www.phind.com/. I find it's much better optimized for coding purposes than regular GPT-4.

Just tried it. I like that it lists the information sources in the right-hand panel, and provides the digested information in the main panel

I've found that it is reasonably competent at most coding tasks; especially on Expert mode.

I rarely have to ask it to alter the code it gives me.

> The big question here is what's next?

These lecture slides from Yann LeCun from 1 month ago. Starting from slide 13, he presents his idea of a roadmap towards autonomous machine intelligence.

https://drive.google.com/file/d/1BU5bV3X5w65DwSMapKcsr0ZvrMR...

I was wondering why nobody combines LLMs with well-known symbolic AI like CYC[1]. Or, perhaps I'm wrong and plenty of companies are working on it.

[1] https://cyc.com/

It depends what you mean by combine. They work in fundamentally different ways. It's a bit like combining an internal combustion engine with a ramjet.

You might be able to compose systems that use LLMs for some sub-components and symbolic AI for other functions. So for example if ChatGPT gets asked a question that LLMs are poor at but symbolic systems are good at, it could switch to using a symbolic system, but that's not really combining the actual technologies.

I’ve had similar thoughts, wondering why LLMs aren’t just a component of a larger system, perhaps used to generate “thoughts” that are then tested by some other subsystems, which in turn may re-fire the LLMs to generate more.

I guess I’m imagining that testing the LLM output might be a problem that some other technique might be suited for, and smells like what we need to mitigate LLM weaknesses.

What I had in mind was to make the output of a natural language parser into symbolic semantic representations (the folks at CYC have such parsers) an additional part of the training data set of the LLM.

The next thing is how do we model quantitative thinking as every facet of what makes us human is involved in quantitative thinking.