Here is a short test of a 7B 4bit model on an intel 8350U laptop with no AMD/Nvidia GPU.

On that laptop CPU from 2017, using a copy of llama.cpp I compiled 2 days ago (just "make", no special options, no BLAS, etc):

  ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "A short test for Hacker News:"

  llama_print_timings:      sample time =    19.12 ms /    36 runs   (    0.53 ms per token,  1882.65 tokens per second)
  llama_print_timings: prompt eval time =   886.82 ms /     9 tokens (   98.54 ms per token,    10.15 tokens per second)
  llama_print_timings:        eval time =  5507.31 ms /    35 runs   (  157.35 ms per token,     6.36 tokens per second)

and a second run:

  ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "Sherlock Holmes favorite dinner was "

  llama_print_timings:      sample time =    54.37 ms /   102 runs   (    0.53 ms per token,  1875.93 tokens per second)
  llama_print_timings: prompt eval time =   876.94 ms /     9 tokens (   97.44 ms per token,    10.26 tokens per second)
  llama_print_timings:        eval time = 16057.95 ms /   101 runs   (  158.99 ms per token,     6.29 tokens per second)

at 158ms per token, if we guess a word is 2.5 tokens, then that's 151 words per minute, much faster than most people can type. On a $250 laptop. Isn't the future neat?

the code I was running: https://github.com/ggerganov/llama.cpp

and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML

There are other models that may perform better, I'm going to be doing a lot of screwing around with OpenLLaMA this weekend.