That's why I wanted to try to understand, what am I missing about local-toy LLMs. How are they not just noise/nonsense generators?
They're bad at generative tasks. Don't have it write code or scientific papers from scratch, but you can have it review anything you've written. You can also do summaries, keyword/entity extraction, and the like safely. Any reductive task works pretty well.
On a whim, I asked Zephyr 7B (Mistral based) “what’s the name of that Ruby method that does <insert code>” and it gave me 3 different correct ways of doing what I wanted, including the one I couldn’t remember. That was a real “oh wow” moment.
So offline situations is the most likely use case for me.
Privacy.
If you use llm's on private documents you don't want others to see, you'll likely prefer local models strongly.
Local might still be there after an online service is no longer available.
There are many times when I am searching for a solution to a problem, and I would be perfectly happy with a possible answer I could test that has a 50% chance of being correct.
Humans should test the output of all AIs.
Humans should test the validity of everything we hear, see, and read.
2. Llama 2
3. Code Llama
4. Orca Mini
5. Vicuna
What can I do with any of these models that won't result in 50% hallucinations/it recommending code with APIs that don't exist/it recommending basically regurgitated StackOverflow historical out of date answers (that it was trained on) for libraries that have had their versions/APIs change, etc?
Can somebody share one real use case they are using any of these models for?