I just tried out the 4-bit version of that model, asking it to write part of a standup comedy routine. It generated some decent ideas, and the speed isn't terrible.

(~3 tokens/second with 20 layers running on the GPU)

I'm using oobabooga in chat-instruct mode. Here's the last few lines of the server log.

  llama_print_timings:        load time =  5310.13 ms
  llama_print_timings:      sample time =   140.31 ms /   200 runs   (    0.70 ms per token)
  llama_print_timings: prompt eval time =  1775.70 ms /    21 tokens (   84.56 ms per token)
  llama_print_timings:        eval time = 47175.08 ms /   199 runs   (  237.06 ms per token)
  llama_print_timings:       total time = 61175.83 ms
  Output generated in 61.48 seconds (3.25 tokens/s, 200 tokens, context 1346, seed 1510260097)
  llama_tokenize: too many tokens
  llama_tokenize: too many tokens
  llama_tokenize: too many tokens
  Llama.generate: prefix-match hit
  
  llama_print_timings:        load time =  5310.13 ms
  llama_print_timings:      sample time =   140.16 ms /   200 runs   (    0.70 ms per token)
  llama_print_timings: prompt eval time = 19554.31 ms /  1740 tokens (   11.24 ms per token)
  llama_print_timings:        eval time = 49292.75 ms /   199 runs   (  247.70 ms per token)
  llama_print_timings:       total time = 81038.76 ms
  Output generated in 81.36 seconds (2.46 tokens/s, 200 tokens, context 1775, seed 1751805530)
  Llama.generate: prefix-match hit
  
  llama_print_timings:        load time =  5310.13 ms
  llama_print_timings:      sample time =   140.75 ms /   200 runs   (    0.70 ms per token)
  llama_print_timings: prompt eval time =  9141.41 ms /   709 tokens (   12.89 ms per token)
  llama_print_timings:        eval time = 49406.39 ms /   199 runs   (  248.27 ms per token)
  llama_print_timings:       total time = 70887.22 ms
  Output generated in 71.18 seconds (2.81 tokens/s, 200 tokens, context 1721, seed 1210059184)