And even in LLMs, it sounds like you're still seeing training/retraining as single monolithic events which are super expensive. Training is super expensive! New techniques (google LoRA; there are others) are changing that.
I hope you get into what has changed in the past few years, starting with Transformer and continuing pretty much daily. There really is a lot of innovation and improvement that is publicly documented and missing from your opinion here.
Long-range Radio??
I do wish people would check if acronyms don't already have well-established meanings before adopting them.
It is a GPT (Using OpenAI's one) and fundamentally it is a black box neural network that when asked to explain it's own decisions, it still cannot transparently do so. This have been an unsolved problem for decades and still overlooked by every AI hype cycle to date.
> And even in LLMs, it sounds like you're still seeing training/retraining as single monolithic events which are super expensive. Training is super expensive! New techniques (google LoRA; there are others) are changing that.
Perhaps that is why only Big Tech companies are the ones doing it for training LLMs and the so-called 'AI companies' aren't doing it themselves and still sit on their AI APIs hence why you could only name Google who can afford to foot the bill for training their LLM.
Everyone else has to sit on their APIs just like before.
> I hope you get into what has changed in the past few years...
I think you have struggled to address the other short-comings that both LLMs and fundamentally neural networks still face that I have already outlined.
Until I see a open-source LLM or some other open source solution that matches or surpasses OpenAI's GPT offerings in parameters, explainability, and is small enough in model size to fit on a smartphone then we can discuss about how this AI cycle is worth the hype. Having yet another AI API product with even larger and expensive models sitting behind a SaaS is an indication of another AI bubble.
Very few pieces of software today explain their decisions, and it doesn't seem to be a big problem for users.
Fundamentally, GPT itself and LLMs are essentially still black-box neural networks and all have the same drawbacks as them:
* More data needed to train them. Which then...
* ...Increases the size of the model (Getting bigger on each revision)
* Opaqueness (Cannot transparently explain its own decisions when it confidently generates the wrong answer.)
* Massive cost involved in training, retraining and fine-tuning it.
The best part is that you can't even give an explanation on why it is generating nonsense or why it overfitted or got confused over an invalid character or input. Little to nothing fundamental about neural networks (which is what LLMs are based on) has changed. It is essentially more of: 'Just waste more money on training it on more data.'
> The Bing openai integration is already dramatically different and better than chatgpt. I’d really advise trying it before forming too many opinions.
What I have said about ChatGPT and GPT-3 still holds true and hyping about a chatbot confidently generating nonsense and that cannot cite its own sources makes it highly untrustworthy to use. It is always eternal to the shortcomings of neural networks fundamentally; which is what LLMs use. They are great sophists and bullshit generators.
At least what needs to exist is an open-source LLM that is more transparent and is smaller than GPT-3 or GPT-4. There is a basic reason why OpenAI won't do that.