Yeah, it's MUCH harder to use because of the lack of tuning.

You have to lean on much older prompt engineering tricks - there are a few initial tips in the LLaMA FAQ here: https://github.com/facebookresearch/llama/blob/main/FAQ.md#2...

Are you getting useful content out of the 7B model? It goes off the rails way too often for me to find it useful.

You might want to tune the sampler. For example, set it to a lower temperature. Also, the 4bit RTN quantisation seems to be messing up the model. Perhaps, the GPTQ quantisation will be much better.

Use `--top_p 2 --top_k 40 --repeat_penalty 1.176 --temp 0.7` with llama.cpp

Not bad with these settings:

    ./main -m ./models/7B/ggml-model-q4_0.bin \
    --top_p 2 --top_k 40 \
    --repeat_penalty 1.176 \
    --temp 0.7 
    -p 'async fn download_url(url: &str)'


    async fn download_url(url: &str) -> io::Result<String> {
      let url = URL(string_value=url);
      if let Some(err) = url.verify() {} // nope, just skip the downloading part
      else match err == None {  // works now
        true => Ok(String::from(match url.open("get")?{
            |res| res.ok().expect_str(&url)?,
            |err: io::Error| Err(io::ErrorKind(uint16_t::MAX as u8))),
            false => Err(io::Error

What fork are you using?

repeat_penalty is not an option.

It is https://github.com/ggerganov/llama.cpp/blob/master/utils.cpp...

It's a new feature :) Pull latest from master.

lol,

    ./main -m ./models/7B/ggml-model-q4_0.bin \
    --top_p 2 --top_k 40 \
    --repeat_penalty 1.176 \
    --temp 0.7
    -p 'To seduce a woman, you first have to'

output:

    import numpy as np
    from scipy.linalg import norm, LinAlgError
    np.random.seed(10)
    x = -2\*norm(LinAlgError())[0]  # error message is too long for command line use
    print x [end of text]

Have you tried using the original repo?

I am not sure that it is just missing instruction fine-tuning. I have been testing with the unicorn prompt from the GPT-2 era and the results are short and not very good. Maybe the sampling parameters in llama.cpp would benefit from some tweaking. For example this is one generation with the 33B model:

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

"Well," said Dr. David Johnson, head of the 17 man/woman team from Oxford, "it's not like we weren't expecting to find them at all."

While many people might be disappointed with their discovery, they were actually quite lucky because very few animals have been found. This makes it nearly impossible to study any sort of creature's behaviour and whatnot. Uncorns are known for being extremely violent so this is definately an exciting time! [end of text]

GPT-3.5 probably has substantially lower cross-entropy loss before instruction fine-tuning as well.

Meta’s LLaMA v. GPT-3 comparisons are to OpenAI’s 2020 release of GPT-3, but there’s been significant progress since then.