In short, my experience is that it's much worse to the point that I won't use LLaMA.

The main change needed seems to be InstructGPT style tuning (https://openai.com/research/instruction-following)

Yeah, it's MUCH harder to use because of the lack of tuning.

You have to lean on much older prompt engineering tricks - there are a few initial tips in the LLaMA FAQ here: https://github.com/facebookresearch/llama/blob/main/FAQ.md#2...

Are you getting useful content out of the 7B model? It goes off the rails way too often for me to find it useful.

You might want to tune the sampler. For example, set it to a lower temperature. Also, the 4bit RTN quantisation seems to be messing up the model. Perhaps, the GPTQ quantisation will be much better.

Use `--top_p 2 --top_k 40 --repeat_penalty 1.176 --temp 0.7` with llama.cpp

Not bad with these settings:

    ./main -m ./models/7B/ggml-model-q4_0.bin \
    --top_p 2 --top_k 40 \
    --repeat_penalty 1.176 \
    --temp 0.7 
    -p 'async fn download_url(url: &str)'


    async fn download_url(url: &str) -> io::Result<String> {
      let url = URL(string_value=url);
      if let Some(err) = url.verify() {} // nope, just skip the downloading part
      else match err == None {  // works now
        true => Ok(String::from(match url.open("get")?{
            |res| res.ok().expect_str(&url)?,
            |err: io::Error| Err(io::ErrorKind(uint16_t::MAX as u8))),
            false => Err(io::Error

What fork are you using?

repeat_penalty is not an option.

It is https://github.com/ggerganov/llama.cpp/blob/master/utils.cpp...

It's a new feature :) Pull latest from master.

lol,

    ./main -m ./models/7B/ggml-model-q4_0.bin \
    --top_p 2 --top_k 40 \
    --repeat_penalty 1.176 \
    --temp 0.7
    -p 'To seduce a woman, you first have to'

output:

    import numpy as np
    from scipy.linalg import norm, LinAlgError
    np.random.seed(10)
    x = -2\*norm(LinAlgError())[0]  # error message is too long for command line use
    print x [end of text]

Have you tried using the original repo?

I am not sure that it is just missing instruction fine-tuning. I have been testing with the unicorn prompt from the GPT-2 era and the results are short and not very good. Maybe the sampling parameters in llama.cpp would benefit from some tweaking. For example this is one generation with the 33B model:

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

"Well," said Dr. David Johnson, head of the 17 man/woman team from Oxford, "it's not like we weren't expecting to find them at all."

While many people might be disappointed with their discovery, they were actually quite lucky because very few animals have been found. This makes it nearly impossible to study any sort of creature's behaviour and whatnot. Uncorns are known for being extremely violent so this is definately an exciting time! [end of text]

GPT-3.5 probably has substantially lower cross-entropy loss before instruction fine-tuning as well.

Meta’s LLaMA v. GPT-3 comparisons are to OpenAI’s 2020 release of GPT-3, but there’s been significant progress since then.

The sampler needs some tuning, but the 65B model has impressive output

https://twitter.com/theshawwn/status/1632569215348531201

Well, gpt3.5-turbo fails the turing test due to it's censorship and legal liability butt covering openai bolted on, so almost anything else is better. Now, compared to openai's gpt3 davinci (text-davinci-003) ... llama is much worse.

I thought LLaMA outscored GPT-3

GPT-3 is a very different model from GPT-3.5. My understanding is that they were comparing LLaMA's performance to benchmark scores published for the original GPT-3, which came out in 2020 and had not yet had instruction tuning, so was significantly harder to use.

I know, that is why I said GPT-3 (Davinci) not GPT-3.5|ChatGPT.

Da Vinci 002 and 003 are actually classified as GPT 3.5 by OpenAI: https://platform.openai.com/docs/models/gpt-3-5

ChatGPT is GPT-3.5 Turbo.

Would you mind summarizing the difference between GPT 3.5 and GPT-3.5 Turbo? I'm not clear about that.

GPT 3.5 is the instruction tuned modern GPT models, such as Da Vinci 002 and 003.

3.5 Turbo is the ChatGPT model: it's cheaper (1/10th the price), faster and has a bunch of extra RLHF training to make it work well as a safe and usable chatbot.

https://openai.com/blog/introducing-chatgpt-and-whisper-apis introduced the turbo model.

Hard to measure these days. The training sets are so large they might contain leaks of test sets. Take these numbers with a grain of salt.

Or... it could be that Chinchilla study has deficiencies in measuring capabilities of models maybe? Either that or your explanation. Frankly I don't think 13B is better than GPT-3 (text-davinci-001 which I think is not RLHF - but maybe better than base)

text-davinci-001 is currently classed as "GPT 3.5" by OpenAI, and it did indeed have RLHF in the form of instruction tuning: https://openai.com/research/instruction-following

MY MISTAKE: 002 and 003 are 3.5, but 001 looks to have pre-dated the InstructGPT work.

I dont know why youre getting downvoted! There is nothing out there at the moment that is as authentic as text-davinci-003. I really hope its not taken away

The writeup includes example text where the algorithm is fed a sentence starting about George Washington and within half a sentence or so goes unhinged and starts praising Trump...

Also, a reminder to folks that this model is not conversationally trained and won't behave like ChatGPT; it cannot take directions.

The original LLaMA paper has some benchmarks. https://arxiv.org/pdf/2302.13971.pdf