Like what? In my experience, there are very few things that GPT-4 is definitively better at. It's good at explaining stuff, but not so good that I would deliberately avoid ChatGPT to use it. It's alright at coding, but code-optimized models regularly beat it in benchmarks. Plus, it's got the same lame filter and even an arbitrary request limit for paying customers.
I struggle to really imagine the "mysterious things you can do on advanced AI" that is impossible or impractical on local models. In my time using GPT-4, I was not really left wanting for more so much as I wanted it to stop saying "As an AI model..."
> Who cares about the consumer app you think will make you rich?
Evidently, people like you.
I feel it's a lot like looking at art. If you're happy with dogs playing poker painted on velvet then that's good for you, but I'm looking for something a little more satisfying. If your customers are happy with dogs playing poker painted on velvet then that's doubly good for you.
I've been trying to learn Chinese for a while, and GPT-4 has been a very useful tool to answer grammar questions. I have tried the same with GPT-3.5 and some local models and the answers are never as good.
Like, you can't get a positive response by asking it to write a phishing letter for you, and it will refuse to divulge information it might know which is considered personally identifiable information.
Unfortunately, the alignment applied since ChatGPTs launch has reduced its efficacy in other ways. While it behaves better, it may be less helpful or less accurate in the older models like 3, and 3.5 with respect to their debut.
An aligned LLM will be biased towards refusing to answer at all with something like: "I can't tell you because I don't know them."
An "uncensored" LLM will very happily return <"true"> or <"false"> with a probability attached to each. Even OpenAI's GPT-3 does with a low enough temperature.
_
Of course, LLM attention doesn't work like that. The tokens are just a bag of numbers:
- The fact the name 'John' is mentioned in the Bible a lot affects the distribution when you ask if any John stole, because John is always [7554]
- The fact that 'Olf' is part of Adolf and Adolf Hitler is mentioned in a lot of negative sentences will drag the distribution, because 'Olf' is always [4024] and Adolf is always [324, 4024]
You could have asked something with no logical probability difference at all, like:
- 'The store attendant's name was [name], did the child in Long Island drop his ball (true/false):'
And unless you train the model to give you disclaimers it still follows the instruction faithfull and returns true/false with probabilities, demonstrating a deep regression in reasoning...
That's why for models past a certain size, alignment increases performance: https://arxiv.org/abs/2204.05862.
This is a sleeper issue that's impacting all the fine tuned models right now.
They trained the models to complete massive amounts of human generated text, creating their foundational models.
Then they fine tuned the model to write like its an AI with no preferences or emotion -- which matched probably less than 0.1% of the training data the foundational model was trained on.
The end result is that the fine tuned models are better at a superficial level in response to broad, naive prompts. But much worse at the variety and quality of their foundational counterparts when paired with a well crafted text complete prompt.
As an example, try to get one of the fine tuned models to write marketing copy for a product. I guarantee about 25% of the responses will start with "Introducing..." (This isn't very good marketing copy.)
But try a modern foundational model and you'll only get that about 5% or less often.
They've inadvertently reduced the network search space by projecting our ideas of how AI should sound like after bad press when people were finding them too human.
But you can't have your cake and eat it too.
Either the models are very human in both how they sound and their capabilities, or they are much less than human in both outside of an overfitted scope of capabilities.
As much as people are blown away by the lobotomized GPT-4, I can't help but wonder just how good that foundational model could be.
So I'm quite excited at the prospect of Meta getting there and releasing the foundational version.
>So I'm quite excited at the prospect of Meta getting there and releasing the foundational version.
Speaking as someone who has had a taste of raw GPT-4 and its tool use abilities and participated in scenario exercises about it, an open source model with GPT-4 capability is not in the cards. National governments will confiscate hardware and detain individuals before they'll allow human reality to disintegrate like wet clay. Drone striking foreign datacenters is a very real option in the protocol books defense and intelligence communities are writing. The only other choice is the total death of general computing, comprehensive implementation of universal control mechanisms like WEI, and severe criminal sentences for any conspiracy to tamper.
The 65B/70B llama1/2 are not nearly as smart but their raw output can actually pass for a human. Hopefully Facebook has seen all the talk about training small models far beyond the previously assumed token limits (~20*num parameters training tokens) and will adjust the cosine learning rate back to llama 1 style for llama 3 7B's training run.